Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (P0_win_upgrade jython_slave) in workspace /data/workspace/temp_centos7_4node_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_centos7_4node_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision d5b2ee1d163aa0ec446b9974f8b68e7fd5f4dab6 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f d5b2ee1d163aa0ec446b9974f8b68e7fd5f4dab6 > /usr/bin/git rev-list d5b2ee1d163aa0ec446b9974f8b68e7fd5f4dab6 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #1904 jenkins-temp_centos7_4node_jython-1904 # timeout=10 No emails were triggered. [temp_centos7_4node_jython] $ /bin/sh -xe /tmp/jenkins3573228106004637937.sh + exit 0 [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins7243671148692326808.sh + jython_bin=/opt/jython/bin + jython_path=/opt/jython/bin/jython + ini_file=/tmp/oel6-4node-rebalance_in_jython.ini + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + echo ' [global] #username:root #password:couchbase username:Administrator password:Membase123 port:8091 n1ql_port:8093 index_port:9102 # data_path:/data/dta # index_path:/data/idx # cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [servers] #1:_1 #2:_2 #3:_3 #4:_4 #5:_5 #6:_6 1:w1 2:w2 3:w3 4:w4 5:w5 6:w6 7:w7 8:w8 9:w9 10:w10 11:w11 12:w12 13:w13 14:w14 15:w15 [_1] ip:172.23.105.155 services: kv [_2] ip:172.23.105.211 [_3] ip:172.23.105.212 [_4] ip:172.23.105.213 [_5] ip:172.23.105.244 [_6] ip:172.23.105.245 [w1] ip:172.23.120.144 [w2] ip:172.23.120.113 [w3] ip:172.23.121.81 [w4] ip:172.23.120.100 [w5] ip:172.23.120.117 [w6] ip:172.23.136.105 [w7] ip:172.23.136.106 [w8] ip:172.23.136.107 [w9] ip:172.23.136.108 [w10] ip:172.23.136.110 [w11] ip:172.23.136.112 [w12] ip:172.23.136.113 [w13] ip:172.23.136.114 [w14] ip:172.23.136.115 [w15] ip:172.23.138.127 [membase] rest_username:Administrator rest_password:password ' + echo 'bucket_collections.travel_sample_app.TravelSampleApp: run_app,initial_load=False,skip_setup_cleanup=True,playbook=rebalance,iteration=5000,monitor_ops_rate=True,log_level=debug ' + rm -rf 'logs/*' jars + '[' '' '!=' '' ']' + sdk_client_params='-P transaction_version=1.1.5 -P client_version=default' + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + '[' false == true ']' + rm -f /tmp/test.pcap + '[' false == true ']' + '[' '-p rerun=False,skip_cleanup=True' == '' ']' + test_params='-p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014' + echo ' activities: - type: "cluster" op_type: "rebalance_in" nodes: 5 service: "kv,kv,n1ql:index,cbas,eventing:fts" - type: "cluster" op_type: "rebalance_out" service: "kv" - type: "cluster" op_type: "rebalance_in" service: "n1ql:index" - type: "cluster" op_type: "rebalance_out" service: "n1ql:index" - type: "cluster" op_type: "rebalance_out" service: "eventing:fts" - type: "cluster" op_type: "rebalance_in" service: "cbas" - type: "cluster" op_type: "rebalance_in" service: "kv" - type: "cluster" op_type: "rebalance_out" service: "cbas" - type: "cluster" op_type: "rebalance_in" service: "eventing:fts" - type: "cluster" nodes: 5 op_type: "rebalance_out" service: "kv,kv,n1ql:index,cbas,eventing:fts" - type: "cluster" op_type: "rebalance_swap" nodes: 3 service: "kv,kv,kv" ' + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.5 -P client_version=default -P 'args=-i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest' Starting a Gradle Daemon, 10 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.5 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: /tmp/test_run.conf Prefix: bucket_collections.travel_sample_app.TravelSampleApp Global Test input params: {'cluster_name': 'oel6-4node-rebalance_in_jython', 'conf_file': '/tmp/test_run.conf', 'ini': '/tmp/oel6-4node-rebalance_in_jython.ini', 'num_nodes': 15, 'rerun': 'False', 'skip_cleanup': 'True', 'spec': 'test_run', 'upgrade_version': '7.0.0-5014'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Apr-26_02-27-37/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/oel6-4node-rebalance_in_jython.ini rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5014 -t bucket_collections.travel_sample_app.TravelSampleApp.run_app,initial_load=False,skip_setup_cleanup=True,playbook=rebalance,iteration=5000,monitor_ops_rate=True,log_level=debug' Test Input params: {'cluster_name': 'oel6-4node-rebalance_in_jython', 'ini': '/tmp/oel6-4node-rebalance_in_jython.ini', 'monitor_ops_rate': 'True', 'log_level': 'debug', 'conf_file': '/tmp/test_run.conf', 'upgrade_version': '7.0.0-5014', 'spec': 'test_run', 'rerun': 'False', 'num_nodes': 15, 'logs_folder': '/data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Apr-26_02-27-37/test_1', 'skip_cleanup': 'True', 'initial_load': 'False', 'iteration': '5000', 'case_number': 1, 'skip_setup_cleanup': 'True', 'playbook': 'rebalance'} run_app (bucket_collections.travel_sample_app.TravelSampleApp) ... 2021-04-26 02:27:59,822 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-04-26 02:28:01,181 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3731] Stats: {u'opsPerSec': 35478.26211094619, u'diskFetches': 77305043, u'dataUsed': 1487874704, u'itemCount': 2718087, u'quotaPercentUsed': 67.50797542432944, u'storageTotals': {u'hdd': {u'usedByData': 2981927851L, u'used': 108967886110L, u'quotaTotal': 223906615296L, u'total': 223906615296L, u'free': 111953307648L}, u'ram': {u'usedByData': 4349168200L, u'used': 13583904768L, u'quotaTotal': 6442450944L, u'total': 32198443008L, u'quotaUsed': 6442450944L, u'quotaUsedPerNode': 2147483648L, u'quotaTotalPerNode': 2147483648L}}, u'diskUsed': 2981927851L, u'vbActiveNumNonResident': 25629, u'memUsed': 4349168200L} 2021-04-26 02:28:01,184 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= AppBase setup started for test #1 run_app ========= 2021-04-26 02:28:01,493 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= AppBase setup complete for test #1 run_app ========= 2021-04-26 02:28:01,494 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-04-26 02:28:01,713 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup complete for test #1 run_app ========= 2021-04-26 02:28:02,105 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:28:02,108 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator]} 2021-04-26 02:28:02,111 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator] 2021-04-26 02:28:02,112 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:28:02,114 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 1 #### 2021-04-26 02:28:02,115 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-26 02:28:02,575 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35478.262111 2021-04-26 02:28:02,782 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:28:02,785 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator]} 2021-04-26 02:28:02,786 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator] 2021-04-26 02:28:02,788 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:28:02,951 | test | DEBUG | pool-2-thread-2 | [rest_client:add_node:1089] Adding remote node 172.23.120.100:8091 to cluster 172.23.120.113:8091 2021-04-26 02:28:05,250 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35478.262111 2021-04-26 02:28:08,101 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36694.578171 2021-04-26 02:28:10,086 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40624.468080 2021-04-26 02:28:12,022 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40624.468080 2021-04-26 02:28:13,884 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40624.468080 2021-04-26 02:28:15,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40624.468080 2021-04-26 02:28:17,882 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41584.396312 2021-04-26 02:28:19,871 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40997.825481 2021-04-26 02:28:21,818 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40997.825481 2021-04-26 02:28:23,826 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40997.825481 2021-04-26 02:28:24,920 | test | DEBUG | pool-2-thread-2 | [rest_client:add_node:1089] Adding remote node 172.23.136.113:8091 to cluster 172.23.120.113:8091 2021-04-26 02:28:26,262 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40997.825481 2021-04-26 02:28:28,535 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39534.213585 2021-04-26 02:28:30,582 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39534.213585 2021-04-26 02:28:32,553 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39534.213585 2021-04-26 02:28:34,581 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39534.213585 2021-04-26 02:28:36,595 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39534.213585 2021-04-26 02:28:38,595 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38562.952388 2021-04-26 02:28:39,892 | test | DEBUG | pool-2-thread-2 | [rest_client:add_node:1089] Adding remote node 172.23.136.114:8091 to cluster 172.23.120.113:8091 2021-04-26 02:28:41,111 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38562.952388 2021-04-26 02:28:42,956 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38562.952388 2021-04-26 02:28:44,982 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38562.952388 2021-04-26 02:28:46,953 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38562.952388 2021-04-26 02:28:48,940 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39193.736456 2021-04-26 02:28:50,997 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39193.736456 2021-04-26 02:28:53,068 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39193.736456 2021-04-26 02:28:55,164 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39193.736456 2021-04-26 02:28:57,079 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39193.736456 2021-04-26 02:28:59,217 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41938.093630 2021-04-26 02:29:00,506 | test | DEBUG | pool-2-thread-2 | [rest_client:add_node:1089] Adding remote node 172.23.136.115:8091 to cluster 172.23.120.113:8091 2021-04-26 02:29:01,515 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41938.093630 2021-04-26 02:29:03,552 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41938.093630 2021-04-26 02:29:05,532 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41938.093630 2021-04-26 02:29:07,490 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40744.049540 2021-04-26 02:29:09,476 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39480.560331 2021-04-26 02:29:11,413 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39480.560331 2021-04-26 02:29:13,328 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39480.560331 2021-04-26 02:29:15,358 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39480.560331 2021-04-26 02:29:17,644 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38812.428024 2021-04-26 02:29:17,743 | test | DEBUG | pool-2-thread-2 | [rest_client:add_node:1089] Adding remote node 172.23.138.127:8091 to cluster 172.23.120.113:8091 2021-04-26 02:29:19,631 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36890.724323 2021-04-26 02:29:21,763 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36890.724323 2021-04-26 02:29:23,676 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36890.724323 2021-04-26 02:29:25,756 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36890.724323 2021-04-26 02:29:26,844 | test | DEBUG | pool-2-thread-2 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': '', 'user': 'Administrator'} 2021-04-26 02:29:26,918 | test | DEBUG | pool-2-thread-2 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:29:26,923 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+------------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------------+-----------------------+---------------+--------------+ | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 4.3875 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 11.8240681328 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 98.2274704578 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 99.3749947916 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 14.2560709345 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 2.29132296822 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 10.3404886504 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 35.7800611682 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 31.9935999467 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 98.7892918106 | Cluster node | | 172.23.120.100 | ['kv'] | | | <--- IN --- | | 172.23.136.113 | ['kv'] | | | <--- IN --- | | 172.23.136.114 | ['n1ql,index'] | | | <--- IN --- | | 172.23.136.115 | ['cbas'] | | | <--- IN --- | | 172.23.138.127 | ['eventing,fts'] | | | <--- IN --- | +----------------+------------------+-----------------------+---------------+--------------+ 2021-04-26 02:29:26,957 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.00 % 2021-04-26 02:29:26,959 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-26 02:29:28,691 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35885.921373 2021-04-26 02:29:32,493 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.05 % 2021-04-26 02:29:32,494 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.05 2021-04-26 02:29:32,798 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36114.951375 2021-04-26 02:29:34,694 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36114.951375 2021-04-26 02:29:36,461 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36114.951375 2021-04-26 02:29:37,746 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.63 % 2021-04-26 02:29:37,747 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.63 2021-04-26 02:29:38,438 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28403.270000 2021-04-26 02:29:40,417 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28403.270000 2021-04-26 02:29:42,012 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28403.270000 2021-04-26 02:29:42,943 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.63 % 2021-04-26 02:29:42,944 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.63 2021-04-26 02:29:43,724 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28403.270000 2021-04-26 02:29:45,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28403.270000 2021-04-26 02:29:47,532 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28618.613069 2021-04-26 02:29:48,115 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.63 % 2021-04-26 02:29:48,115 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.63 2021-04-26 02:29:49,553 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20878.153362 2021-04-26 02:29:51,545 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20878.153362 2021-04-26 02:29:53,338 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.94 % 2021-04-26 02:29:53,338 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.94 2021-04-26 02:29:53,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20878.153362 2021-04-26 02:29:55,354 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20878.153362 2021-04-26 02:29:57,358 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22908.453669 2021-04-26 02:29:58,569 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:29:58,571 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:29:59,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25504.047779 2021-04-26 02:30:01,092 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25504.047779 2021-04-26 02:30:03,040 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25504.047779 2021-04-26 02:30:03,752 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:30:03,753 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:30:04,823 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25504.047779 2021-04-26 02:30:06,507 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25966.560113 2021-04-26 02:30:08,359 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26146.608978 2021-04-26 02:30:08,938 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:30:08,940 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:30:10,174 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26177.364979 2021-04-26 02:30:12,055 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26177.364979 2021-04-26 02:30:13,924 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26177.364979 2021-04-26 02:30:14,105 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:30:14,105 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:30:15,647 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26177.364979 2021-04-26 02:30:17,530 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26345.214876 2021-04-26 02:30:19,283 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:30:19,283 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:30:19,349 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28754.493870 2021-04-26 02:30:21,111 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28754.493870 2021-04-26 02:30:22,736 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28754.493870 2021-04-26 02:30:24,471 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.31 % 2021-04-26 02:30:24,471 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.31 2021-04-26 02:30:24,760 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28754.493870 2021-04-26 02:30:26,454 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26611.069608 2021-04-26 02:30:28,220 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20202.551795 2021-04-26 02:30:29,645 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:29,647 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:30,029 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20202.551795 2021-04-26 02:30:31,828 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20202.551795 2021-04-26 02:30:33,744 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20202.551795 2021-04-26 02:30:34,823 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:34,825 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:35,460 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20202.551795 2021-04-26 02:30:37,259 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22393.802006 2021-04-26 02:30:39,085 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27666.877639 2021-04-26 02:30:39,974 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:39,976 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:40,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27666.877639 2021-04-26 02:30:42,727 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27666.877639 2021-04-26 02:30:44,733 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27666.877639 2021-04-26 02:30:45,130 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:45,131 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:46,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27666.877639 2021-04-26 02:30:48,263 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30956.121073 2021-04-26 02:30:50,068 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30956.121073 2021-04-26 02:30:50,256 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:50,256 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:52,007 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30956.121073 2021-04-26 02:30:53,842 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30956.121073 2021-04-26 02:30:55,407 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:30:55,407 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:30:55,690 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30956.121073 2021-04-26 02:30:57,517 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28411.574465 2021-04-26 02:30:59,348 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28411.574465 2021-04-26 02:31:00,608 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:00,609 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:01,157 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28411.574465 2021-04-26 02:31:02,993 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28411.574465 2021-04-26 02:31:04,956 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28411.574465 2021-04-26 02:31:05,779 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:05,779 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:06,740 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28286.461954 2021-04-26 02:31:08,594 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26455.087864 2021-04-26 02:31:10,519 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26455.087864 2021-04-26 02:31:10,917 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:10,917 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:12,391 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26455.087864 2021-04-26 02:31:14,229 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26455.087864 2021-04-26 02:31:15,947 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26455.087864 2021-04-26 02:31:16,115 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:16,115 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:17,805 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26651.921799 2021-04-26 02:31:19,690 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26248.234221 2021-04-26 02:31:21,289 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:21,289 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:21,815 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26248.234221 2021-04-26 02:31:23,681 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26248.234221 2021-04-26 02:31:25,335 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26248.234221 2021-04-26 02:31:26,480 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:26,482 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:26,986 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27235.848481 2021-04-26 02:31:28,651 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27235.848481 2021-04-26 02:31:30,509 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28447.462949 2021-04-26 02:31:31,605 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:31,605 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:32,628 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28447.462949 2021-04-26 02:31:34,520 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28447.462949 2021-04-26 02:31:36,207 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28447.462949 2021-04-26 02:31:36,753 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:36,753 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:38,078 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22879.368837 2021-04-26 02:31:39,994 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22879.368837 2021-04-26 02:31:41,872 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22879.368837 2021-04-26 02:31:41,930 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:41,930 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:43,813 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22879.368837 2021-04-26 02:31:45,460 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22879.368837 2021-04-26 02:31:47,121 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:47,124 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:47,325 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25297.197435 2021-04-26 02:31:48,980 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27507.138404 2021-04-26 02:31:50,857 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27507.138404 2021-04-26 02:31:52,301 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:52,302 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:52,710 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27507.138404 2021-04-26 02:31:54,519 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27507.138404 2021-04-26 02:31:56,184 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27507.138404 2021-04-26 02:31:57,463 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:31:57,463 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:31:58,073 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22768.088392 2021-04-26 02:31:59,934 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22768.088392 2021-04-26 02:32:02,211 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22768.088392 2021-04-26 02:32:02,630 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.36 % 2021-04-26 02:32:02,630 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.36 2021-04-26 02:32:04,045 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22768.088392 2021-04-26 02:32:05,707 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22768.088392 2021-04-26 02:32:07,632 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25471.517851 2021-04-26 02:32:07,812 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:07,812 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:09,523 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25471.517851 2021-04-26 02:32:11,463 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25471.517851 2021-04-26 02:32:12,990 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:12,990 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:13,411 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25471.517851 2021-04-26 02:32:15,322 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25471.517851 2021-04-26 02:32:17,176 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25564.867319 2021-04-26 02:32:18,164 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:18,164 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:19,171 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27898.944828 2021-04-26 02:32:21,124 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27898.944828 2021-04-26 02:32:23,115 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27898.944828 2021-04-26 02:32:23,338 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:23,338 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:24,956 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27898.944828 2021-04-26 02:32:26,720 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28049.101693 2021-04-26 02:32:28,497 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:28,499 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:28,556 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28136.959090 2021-04-26 02:32:30,176 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28136.959090 2021-04-26 02:32:32,030 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28136.959090 2021-04-26 02:32:33,648 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:33,648 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:33,703 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28136.959090 2021-04-26 02:32:35,426 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28136.959090 2021-04-26 02:32:37,230 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24324.192739 2021-04-26 02:32:38,806 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:38,808 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:39,127 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15000.251773 2021-04-26 02:32:41,017 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15000.251773 2021-04-26 02:32:42,976 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15000.251773 2021-04-26 02:32:43,984 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:43,986 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:44,855 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15000.251773 2021-04-26 02:32:46,530 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15338.069135 2021-04-26 02:32:48,328 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25491.105526 2021-04-26 02:32:49,150 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:49,151 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:50,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25491.105526 2021-04-26 02:32:51,894 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25491.105526 2021-04-26 02:32:53,711 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25491.105526 2021-04-26 02:32:54,312 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:54,313 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:32:55,407 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25491.105526 2021-04-26 02:32:57,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24615.051004 2021-04-26 02:32:59,073 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22710.612664 2021-04-26 02:32:59,480 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:32:59,480 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:00,950 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22710.612664 2021-04-26 02:33:02,852 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22710.612664 2021-04-26 02:33:04,641 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:04,641 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:04,687 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22710.612664 2021-04-26 02:33:06,437 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22867.250699 2021-04-26 02:33:08,255 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30649.497987 2021-04-26 02:33:09,821 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:09,822 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:10,131 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30649.497987 2021-04-26 02:33:11,989 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30649.497987 2021-04-26 02:33:13,858 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30649.497987 2021-04-26 02:33:14,990 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:14,990 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:15,548 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30649.497987 2021-04-26 02:33:17,357 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30811.258602 2021-04-26 02:33:19,190 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31724.047233 2021-04-26 02:33:20,144 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:20,145 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:21,092 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31724.047233 2021-04-26 02:33:22,908 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31724.047233 2021-04-26 02:33:24,703 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31724.047233 2021-04-26 02:33:25,345 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:25,345 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:26,444 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31664.341262 2021-04-26 02:33:28,411 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28744.129965 2021-04-26 02:33:30,275 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28744.129965 2021-04-26 02:33:30,487 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:30,487 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:32,131 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28744.129965 2021-04-26 02:33:33,877 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28744.129965 2021-04-26 02:33:35,464 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28744.129965 2021-04-26 02:33:35,642 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:35,644 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:37,138 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28547.934156 2021-04-26 02:33:39,071 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23933.376094 2021-04-26 02:33:40,769 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:40,770 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:41,030 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23933.376094 2021-04-26 02:33:42,934 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23933.376094 2021-04-26 02:33:44,890 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23933.376094 2021-04-26 02:33:45,930 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:45,930 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:46,717 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24337.955412 2021-04-26 02:33:48,552 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27417.126159 2021-04-26 02:33:50,486 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27417.126159 2021-04-26 02:33:51,102 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:51,102 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:52,313 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27417.126159 2021-04-26 02:33:54,151 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27417.126159 2021-04-26 02:33:56,000 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27417.126159 2021-04-26 02:33:56,253 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:33:56,253 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:33:57,842 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28410.244952 2021-04-26 02:33:59,674 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31845.939914 2021-04-26 02:34:01,417 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:01,417 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:01,509 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31845.939914 2021-04-26 02:34:03,398 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31845.939914 2021-04-26 02:34:05,243 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31845.939914 2021-04-26 02:34:06,601 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:06,601 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:07,101 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32040.245549 2021-04-26 02:34:08,953 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27975.929354 2021-04-26 02:34:10,816 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27975.929354 2021-04-26 02:34:11,769 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:11,770 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:12,753 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27975.929354 2021-04-26 02:34:14,573 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27975.929354 2021-04-26 02:34:16,364 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26770.578999 2021-04-26 02:34:16,957 | test | DEBUG | pool-2-thread-2 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:16,957 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:17,980 | test | INFO | pool-2-thread-2 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.120.113: 2021-04-26 02:34:17,980 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429368961L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.961Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-26 02:34:17,982 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619429368691L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.691Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.113\' in 0 seconds.'} 2021-04-26 02:34:17,983 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368596L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.596Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.120.100\' in 0 seconds.'} 2021-04-26 02:34:17,983 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368095L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.095Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-26 02:34:17,983 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429367256L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:27.256Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-26 02:34:17,983 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366879L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.879Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-04-26 02:34:17,984 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366599L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.599Z', u'text': u'Node ns_1@172.23.138.127 joined cluster'} 2021-04-26 02:34:17,984 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366347L, u'shortText': u'web start ok', u'serverTime': u'2021-04-26T02:29:26.347Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.138.127\'. Version: "7.0.0-5014-enterprise".'} 2021-04-26 02:34:17,986 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429366084L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.084Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.120.100','ns_1@172.23.136.114',\n 'ns_1@172.23.136.106','ns_1@172.23.136.107',\n 'ns_1@172.23.120.113','ns_1@172.23.120.117',\n 'ns_1@172.23.138.127','ns_1@172.23.121.81',\n 'ns_1@172.23.120.144','ns_1@172.23.136.108',\n 'ns_1@172.23.136.112','ns_1@172.23.136.115',\n 'ns_1@172.23.136.113','ns_1@172.23.136.110',\n 'ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = bf79f21e3f319411b95b77583f9b47a3"} 2021-04-26 02:34:17,986 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.136.114', u'tstamp': 1619429360969L, u'shortText': u'node up', u'serverTime': u'2021-04-26T02:29:20.969Z', u'text': u"Node 'ns_1@172.23.136.114' saw that node 'ns_1@172.23.138.127' came up. Tags: []"} 2021-04-26 02:34:17,989 | test | ERROR | pool-2-thread-2 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-04-26 02:34:17,990 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-26 02:34:18,328 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28176.177966 2021-04-26 02:34:18,444 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:34:18,447 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:34:18,447 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:34:18,447 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:34:18,447 | test | DEBUG | MainThread | [cluster:scenario_rebalance_out:87] Removing node with services: kv 2021-04-26 02:34:20,237 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28176.177966 2021-04-26 02:34:20,819 | test | DEBUG | pool-2-thread-3 | [task:start_rebalance:326] Removing node 172.23.120.113:8091 from cluster 2021-04-26 02:34:20,819 | test | DEBUG | pool-2-thread-3 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': u'ns_1@172.23.120.113', 'user': 'Administrator'} 2021-04-26 02:34:20,839 | test | DEBUG | pool-2-thread-3 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:34:20,844 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 52.4236490146 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 1.48416666667 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 16.4195895103 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 14.2451187093 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 96.3149385823 | --- OUT ---> | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 96.4982749712 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 2.55170919515 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 39.5854827437 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 1.25084375703 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 11.742597855 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 28.9697585853 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 1.24916666667 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 58.0081168696 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 20.7551729598 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 94.4172265576 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-26 02:34:20,874 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:20,874 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:22,144 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28176.177966 2021-04-26 02:34:24,035 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28176.177966 2021-04-26 02:34:25,717 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28176.177966 2021-04-26 02:34:26,058 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:26,059 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:27,632 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29891.847444 2021-04-26 02:34:29,563 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29891.847444 2021-04-26 02:34:31,226 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:31,227 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:31,431 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29891.847444 2021-04-26 02:34:33,270 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29891.847444 2021-04-26 02:34:35,131 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29891.847444 2021-04-26 02:34:36,410 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:36,411 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:36,907 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29721.684998 2021-04-26 02:34:38,592 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31002.521446 2021-04-26 02:34:40,266 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31002.521446 2021-04-26 02:34:41,582 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.40 % 2021-04-26 02:34:41,582 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.4 2021-04-26 02:34:41,996 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31002.521446 2021-04-26 02:34:43,871 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31002.521446 2021-04-26 02:34:45,585 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31002.521446 2021-04-26 02:34:46,762 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.50 % 2021-04-26 02:34:46,763 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.5 2021-04-26 02:34:47,631 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22949.712458 2021-04-26 02:34:49,581 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20223.057789 2021-04-26 02:34:51,513 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20223.057789 2021-04-26 02:34:51,943 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 1.92 % 2021-04-26 02:34:51,943 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.92 2021-04-26 02:34:53,438 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20223.057789 2021-04-26 02:34:55,265 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20223.057789 2021-04-26 02:34:57,119 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23202.286224 2021-04-26 02:34:57,144 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.23 % 2021-04-26 02:34:57,144 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.23 2021-04-26 02:34:59,015 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27597.111619 2021-04-26 02:35:00,881 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27597.111619 2021-04-26 02:35:02,312 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:02,312 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:02,770 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27597.111619 2021-04-26 02:35:04,598 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27597.111619 2021-04-26 02:35:06,480 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27803.703090 2021-04-26 02:35:07,500 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:07,500 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:08,444 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25885.934435 2021-04-26 02:35:10,321 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25885.934435 2021-04-26 02:35:12,227 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25885.934435 2021-04-26 02:35:12,691 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:12,691 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:14,099 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25885.934435 2021-04-26 02:35:15,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25885.934435 2021-04-26 02:35:17,819 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29593.291932 2021-04-26 02:35:17,892 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:17,894 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:19,631 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29593.291932 2021-04-26 02:35:21,480 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29593.291932 2021-04-26 02:35:23,091 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:23,091 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:23,303 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29593.291932 2021-04-26 02:35:25,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29593.291932 2021-04-26 02:35:27,075 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29363.007643 2021-04-26 02:35:28,275 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:28,275 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:28,907 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28509.750852 2021-04-26 02:35:30,769 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28509.750852 2021-04-26 02:35:32,720 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28509.750852 2021-04-26 02:35:33,473 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:33,473 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:34,565 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28509.750852 2021-04-26 02:35:36,273 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28509.750852 2021-04-26 02:35:38,176 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29434.713196 2021-04-26 02:35:38,653 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:38,654 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:40,056 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29434.713196 2021-04-26 02:35:41,819 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29434.713196 2021-04-26 02:35:43,578 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29434.713196 2021-04-26 02:35:43,825 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:43,825 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:45,342 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29434.713196 2021-04-26 02:35:47,203 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27305.787029 2021-04-26 02:35:48,973 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:48,973 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:49,115 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 21717.541642 2021-04-26 02:35:50,974 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 21717.541642 2021-04-26 02:35:53,007 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 21717.541642 2021-04-26 02:35:54,167 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:54,168 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:35:54,976 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 21717.541642 2021-04-26 02:35:56,867 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 21834.792234 2021-04-26 02:35:58,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29438.062274 2021-04-26 02:35:59,351 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:35:59,351 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:00,569 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29438.062274 2021-04-26 02:36:02,559 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29438.062274 2021-04-26 02:36:04,448 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29438.062274 2021-04-26 02:36:04,519 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:04,519 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:06,176 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29438.062274 2021-04-26 02:36:08,036 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27979.836756 2021-04-26 02:36:09,706 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:09,707 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:09,986 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23131.839459 2021-04-26 02:36:11,933 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23131.839459 2021-04-26 02:36:13,845 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23131.839459 2021-04-26 02:36:14,880 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:14,880 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:15,614 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23131.839459 2021-04-26 02:36:17,470 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26896.041706 2021-04-26 02:36:19,339 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27149.292261 2021-04-26 02:36:20,066 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:20,066 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:21,167 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27149.292261 2021-04-26 02:36:22,996 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27149.292261 2021-04-26 02:36:24,796 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27149.292261 2021-04-26 02:36:25,246 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:25,246 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:26,605 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27149.292261 2021-04-26 02:36:28,453 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25795.053739 2021-04-26 02:36:30,365 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25795.053739 2021-04-26 02:36:30,405 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:30,407 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:32,413 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25795.053739 2021-04-26 02:36:34,466 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25795.053739 2021-04-26 02:36:35,596 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:35,596 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:36,197 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25795.053739 2021-04-26 02:36:38,194 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26339.703676 2021-04-26 02:36:40,122 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28383.378657 2021-04-26 02:36:40,773 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:40,773 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:41,933 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28383.378657 2021-04-26 02:36:43,812 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28383.378657 2021-04-26 02:36:45,510 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28383.378657 2021-04-26 02:36:45,951 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:45,953 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:47,257 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26793.483669 2021-04-26 02:36:48,954 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24533.764556 2021-04-26 02:36:50,937 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24533.764556 2021-04-26 02:36:51,128 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:51,128 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:52,806 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24533.764556 2021-04-26 02:36:54,736 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24533.764556 2021-04-26 02:36:56,316 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:36:56,316 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:36:56,424 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24533.764556 2021-04-26 02:36:58,318 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22269.210962 2021-04-26 02:37:00,276 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22269.210962 2021-04-26 02:37:01,497 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:01,499 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:02,217 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22269.210962 2021-04-26 02:37:03,914 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22269.210962 2021-04-26 02:37:05,753 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22269.210962 2021-04-26 02:37:06,677 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:06,677 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:07,660 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22426.332466 2021-04-26 02:37:09,539 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23571.413288 2021-04-26 02:37:11,530 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23571.413288 2021-04-26 02:37:11,851 | test | DEBUG | pool-2-thread-3 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:11,851 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:12,296 | test | INFO | pool-2-thread-3 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.120.100: 2021-04-26 02:37:12,296 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429660304L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:34:20.304Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-04-26 02:37:12,296 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429368961L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.961Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-26 02:37:12,298 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619429368691L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.691Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.113\' in 0 seconds.'} 2021-04-26 02:37:12,298 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368596L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.596Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.120.100\' in 0 seconds.'} 2021-04-26 02:37:12,299 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368095L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.095Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-26 02:37:12,299 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429367256L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:27.256Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-26 02:37:12,299 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366879L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.879Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-04-26 02:37:12,299 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366599L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.599Z', u'text': u'Node ns_1@172.23.138.127 joined cluster'} 2021-04-26 02:37:12,299 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366347L, u'shortText': u'web start ok', u'serverTime': u'2021-04-26T02:29:26.347Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.138.127\'. Version: "7.0.0-5014-enterprise".'} 2021-04-26 02:37:12,301 | test | ERROR | pool-2-thread-3 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429366084L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.084Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.120.100','ns_1@172.23.136.114',\n 'ns_1@172.23.136.106','ns_1@172.23.136.107',\n 'ns_1@172.23.120.113','ns_1@172.23.120.117',\n 'ns_1@172.23.138.127','ns_1@172.23.121.81',\n 'ns_1@172.23.120.144','ns_1@172.23.136.108',\n 'ns_1@172.23.136.112','ns_1@172.23.136.115',\n 'ns_1@172.23.136.113','ns_1@172.23.136.110',\n 'ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = bf79f21e3f319411b95b77583f9b47a3"} 2021-04-26 02:37:12,301 | test | ERROR | pool-2-thread-3 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-04-26 02:37:12,302 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-26 02:37:12,688 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:37:12,690 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:37:12,690 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:37:12,690 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:37:13,546 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23571.413288 2021-04-26 02:37:14,898 | test | DEBUG | pool-2-thread-4 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': '', 'user': 'Administrator'} 2021-04-26 02:37:14,923 | test | DEBUG | pool-2-thread-4 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:37:14,924 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 53.0858333333 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 0.741672847274 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 10.327241819 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 12.1615653203 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 97.3816666667 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 98.8533142219 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 1.37753443836 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 19.1155147414 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 4.27242879285 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 10.9616666667 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 36.6788610095 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 1.3275 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 55.1044557219 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 19.0159829738 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 96.4058932351 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-26 02:37:14,950 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:14,950 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:15,334 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23571.413288 2021-04-26 02:37:17,309 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 23571.413288 2021-04-26 02:37:19,286 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27521.679065 2021-04-26 02:37:20,157 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:20,157 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:21,137 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27521.679065 2021-04-26 02:37:23,062 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27521.679065 2021-04-26 02:37:24,977 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27521.679065 2021-04-26 02:37:25,338 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:25,339 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:26,684 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27521.679065 2021-04-26 02:37:28,644 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28291.701693 2021-04-26 02:37:30,487 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28291.701693 2021-04-26 02:37:30,509 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:30,509 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:32,410 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28291.701693 2021-04-26 02:37:34,233 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28291.701693 2021-04-26 02:37:35,674 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:35,674 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:35,924 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28291.701693 2021-04-26 02:37:37,819 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:39,720 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:40,844 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:40,845 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:41,671 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:43,505 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:45,286 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:46,039 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:46,039 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:47,059 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26934.811872 2021-04-26 02:37:48,874 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25124.262173 2021-04-26 02:37:50,596 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25124.262173 2021-04-26 02:37:51,214 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:51,216 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:52,319 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25124.262173 2021-04-26 02:37:54,092 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25124.262173 2021-04-26 02:37:55,816 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25124.262173 2021-04-26 02:37:56,377 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:37:56,378 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:37:57,796 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:37:59,742 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:38:01,545 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:01,546 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:01,676 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:38:03,461 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:38:05,230 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:38:06,721 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:06,721 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:06,881 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17384.353637 2021-04-26 02:38:08,667 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18422.112624 2021-04-26 02:38:10,421 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18422.112624 2021-04-26 02:38:11,882 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:11,884 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:12,173 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18422.112624 2021-04-26 02:38:13,924 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18422.112624 2021-04-26 02:38:15,618 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18422.112624 2021-04-26 02:38:17,069 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:17,069 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:17,437 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13601.038008 2021-04-26 02:38:19,220 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12502.934215 2021-04-26 02:38:20,963 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12502.934215 2021-04-26 02:38:22,243 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:22,244 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:22,737 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12502.934215 2021-04-26 02:38:24,506 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12502.934215 2021-04-26 02:38:26,233 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12502.934215 2021-04-26 02:38:27,431 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:27,433 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:28,039 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:29,822 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:31,588 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:32,609 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:32,611 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:33,359 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:35,092 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:36,890 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13519.473796 2021-04-26 02:38:37,779 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:37,779 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:38,726 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14087.088917 2021-04-26 02:38:40,532 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14087.088917 2021-04-26 02:38:42,315 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14087.088917 2021-04-26 02:38:42,956 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:42,956 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:44,122 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14087.088917 2021-04-26 02:38:45,857 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14087.088917 2021-04-26 02:38:47,602 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:48,122 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:48,124 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:49,301 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:51,028 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:52,769 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:53,290 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:53,292 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:54,480 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:56,233 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12320.004759 2021-04-26 02:38:57,937 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:38:58,456 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:38:58,457 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:38:59,664 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:39:01,388 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:39:03,101 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:39:03,641 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:03,642 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:04,815 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:39:06,596 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9577.085535 2021-04-26 02:39:08,306 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:08,801 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:08,801 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:09,986 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:11,704 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:13,424 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:13,956 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:13,957 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:15,131 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:16,867 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8893.891333 2021-04-26 02:39:18,595 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8579.731571 2021-04-26 02:39:19,127 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:19,128 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:20,321 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8579.731571 2021-04-26 02:39:22,062 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8579.731571 2021-04-26 02:39:23,798 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8579.731571 2021-04-26 02:39:24,299 | test | DEBUG | pool-2-thread-4 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:24,301 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:24,703 | test | INFO | pool-2-thread-4 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.120.100: 2021-04-26 02:39:24,703 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429834803L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:37:14.803Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-04-26 02:39:24,704 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429660304L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:34:20.304Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-04-26 02:39:24,704 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429368961L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.961Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-26 02:39:24,704 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619429368691L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.691Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.113\' in 0 seconds.'} 2021-04-26 02:39:24,706 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368596L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.596Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.120.100\' in 0 seconds.'} 2021-04-26 02:39:24,706 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.120.100', u'tstamp': 1619429368095L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:28.095Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-26 02:39:24,706 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.113', u'tstamp': 1619429367256L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:27.256Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-26 02:39:24,707 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366879L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.879Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-04-26 02:39:24,707 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366599L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:29:26.599Z', u'text': u'Node ns_1@172.23.138.127 joined cluster'} 2021-04-26 02:39:24,707 | test | ERROR | pool-2-thread-4 | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619429366347L, u'shortText': u'web start ok', u'serverTime': u'2021-04-26T02:29:26.347Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.138.127\'. Version: "7.0.0-5014-enterprise".'} 2021-04-26 02:39:24,709 | test | ERROR | pool-2-thread-4 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-04-26 02:39:24,709 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-26 02:39:25,053 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:39:25,055 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:39:25,055 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:39:25,055 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:39:25,055 | test | DEBUG | MainThread | [cluster:scenario_rebalance_out:87] Removing node with services: n1ql,index 2021-04-26 02:39:25,522 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8579.731571 2021-04-26 02:39:27,259 | test | DEBUG | pool-2-thread-5 | [task:start_rebalance:326] Removing node 172.23.121.81:8091 from cluster 2021-04-26 02:39:27,259 | test | DEBUG | pool-2-thread-5 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': u'ns_1@172.23.121.81', 'user': 'Administrator'} 2021-04-26 02:39:27,259 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8649.805377 2021-04-26 02:39:27,276 | test | DEBUG | pool-2-thread-5 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:39:27,280 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 39.2455062874 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 0.663344389073 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 35.5602963358 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 1.00335005583 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 95.1181706057 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 90.7294756841 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 1.41916666667 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 24.856040467 | --- OUT ---> | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 2.3707938201 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 2.17579707005 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 36.9411411762 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 0.484154562803 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 48.2941950967 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 22.2786908579 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 79.7925 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-26 02:39:27,305 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:27,305 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:28,959 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11383.630117 2021-04-26 02:39:30,572 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11383.630117 2021-04-26 02:39:32,357 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11383.630117 2021-04-26 02:39:32,483 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:32,484 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:34,147 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11383.630117 2021-04-26 02:39:35,868 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11383.630117 2021-04-26 02:39:37,617 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:37,644 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:37,645 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:39,355 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:41,056 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:42,773 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:42,813 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 2.50 % 2021-04-26 02:39:42,813 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-04-26 02:39:44,798 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:46,780 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10210.316193 2021-04-26 02:39:48,039 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 7.67 % 2021-04-26 02:39:48,039 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 7.67 2021-04-26 02:39:48,960 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10377.679609 2021-04-26 02:39:51,000 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10377.679609 2021-04-26 02:39:53,137 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10377.679609 2021-04-26 02:39:53,316 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 9.40 % 2021-04-26 02:39:53,316 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 9.4 2021-04-26 02:39:55,134 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10377.679609 2021-04-26 02:39:56,884 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10377.679609 2021-04-26 02:39:58,479 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 10.29 % 2021-04-26 02:39:58,480 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 10.29 2021-04-26 02:39:58,752 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12574.940592 2021-04-26 02:40:00,477 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12574.940592 2021-04-26 02:40:02,411 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12574.940592 2021-04-26 02:40:03,673 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 11.75 % 2021-04-26 02:40:03,674 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 11.75 2021-04-26 02:40:04,489 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12574.940592 2021-04-26 02:40:06,355 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12574.940592 2021-04-26 02:40:08,415 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9203.537558 2021-04-26 02:40:08,930 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 14.20 % 2021-04-26 02:40:08,931 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 14.2 2021-04-26 02:40:10,358 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9203.537558 2021-04-26 02:40:12,266 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9203.537558 2021-04-26 02:40:14,154 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 15.65 % 2021-04-26 02:40:14,154 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 15.65 2021-04-26 02:40:14,250 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9203.537558 2021-04-26 02:40:16,061 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9203.537558 2021-04-26 02:40:18,134 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9094.931037 2021-04-26 02:40:19,388 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 17.60 % 2021-04-26 02:40:19,388 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 17.6 2021-04-26 02:40:20,134 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10167.855916 2021-04-26 02:40:22,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10167.855916 2021-04-26 02:40:24,174 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10167.855916 2021-04-26 02:40:24,598 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 19.52 % 2021-04-26 02:40:24,598 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 19.52 2021-04-26 02:40:26,115 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10167.855916 2021-04-26 02:40:28,019 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10099.644591 2021-04-26 02:40:29,778 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 21.33 % 2021-04-26 02:40:29,778 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 21.33 2021-04-26 02:40:29,983 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8867.743799 2021-04-26 02:40:31,933 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8867.743799 2021-04-26 02:40:33,832 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8867.743799 2021-04-26 02:40:34,986 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 23.45 % 2021-04-26 02:40:34,987 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 23.45 2021-04-26 02:40:35,706 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8867.743799 2021-04-26 02:40:37,546 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7387.862519 2021-04-26 02:40:39,469 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7387.862519 2021-04-26 02:40:40,187 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 25.21 % 2021-04-26 02:40:40,187 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 25.21 2021-04-26 02:40:41,351 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7387.862519 2021-04-26 02:40:43,272 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7387.862519 2021-04-26 02:40:45,398 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7387.862519 2021-04-26 02:40:45,418 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 27.07 % 2021-04-26 02:40:45,418 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 27.07 2021-04-26 02:40:47,579 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7430.774489 2021-04-26 02:40:49,641 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6838.969002 2021-04-26 02:40:50,772 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 28.90 % 2021-04-26 02:40:50,772 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 28.9 2021-04-26 02:40:51,608 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6838.969002 2021-04-26 02:40:53,575 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6838.969002 2021-04-26 02:40:55,446 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6838.969002 2021-04-26 02:40:55,996 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 30.74 % 2021-04-26 02:40:55,996 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.74 2021-04-26 02:40:57,540 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6971.481980 2021-04-26 02:40:59,378 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7203.075647 2021-04-26 02:41:01,145 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7203.075647 2021-04-26 02:41:01,157 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 31.77 % 2021-04-26 02:41:01,157 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.77 2021-04-26 02:41:02,878 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7203.075647 2021-04-26 02:41:04,687 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7203.075647 2021-04-26 02:41:06,344 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 33.33 % 2021-04-26 02:41:06,344 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-26 02:41:06,766 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7203.075647 2021-04-26 02:41:08,536 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5485.047374 2021-04-26 02:41:10,256 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5485.047374 2021-04-26 02:41:11,569 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 33.33 % 2021-04-26 02:41:11,569 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-26 02:41:11,950 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5485.047374 2021-04-26 02:41:13,641 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5485.047374 2021-04-26 02:41:15,329 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5485.047374 2021-04-26 02:41:16,724 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 33.33 % 2021-04-26 02:41:16,724 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-26 02:41:17,022 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6117.635456 2021-04-26 02:41:18,783 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8703.140025 2021-04-26 02:41:20,502 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8703.140025 2021-04-26 02:41:21,881 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 43.33 % 2021-04-26 02:41:21,881 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.33 2021-04-26 02:41:22,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8703.140025 2021-04-26 02:41:23,930 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8703.140025 2021-04-26 02:41:25,642 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8703.140025 2021-04-26 02:41:27,022 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 43.33 % 2021-04-26 02:41:27,022 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.33 2021-04-26 02:41:27,348 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:29,052 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:30,865 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:32,184 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.33 % 2021-04-26 02:41:32,184 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.33 2021-04-26 02:41:32,759 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:34,694 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:36,413 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14664.655665 2021-04-26 02:41:37,365 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.33 % 2021-04-26 02:41:37,365 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.33 2021-04-26 02:41:38,088 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14683.960833 2021-04-26 02:41:39,996 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14683.960833 2021-04-26 02:41:41,947 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14683.960833 2021-04-26 02:41:42,490 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.33 % 2021-04-26 02:41:42,490 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.33 2021-04-26 02:41:43,733 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14683.960833 2021-04-26 02:41:45,477 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14683.960833 2021-04-26 02:41:47,344 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13219.220394 2021-04-26 02:41:47,628 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.33 % 2021-04-26 02:41:47,628 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.33 2021-04-26 02:41:49,269 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10389.313953 2021-04-26 02:41:51,142 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10389.313953 2021-04-26 02:41:52,839 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.52 % 2021-04-26 02:41:52,841 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.52 2021-04-26 02:41:52,967 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10389.313953 2021-04-26 02:41:54,825 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10389.313953 2021-04-26 02:41:56,640 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10389.313953 2021-04-26 02:41:58,049 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.78 % 2021-04-26 02:41:58,051 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.78 2021-04-26 02:41:58,413 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11508.197611 2021-04-26 02:42:00,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11508.197611 2021-04-26 02:42:01,969 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11508.197611 2021-04-26 02:42:03,250 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.83 % 2021-04-26 02:42:03,250 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.83 2021-04-26 02:42:03,730 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11508.197611 2021-04-26 02:42:05,516 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11508.197611 2021-04-26 02:42:07,237 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 9144.909281 2021-04-26 02:42:08,428 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.89 % 2021-04-26 02:42:08,430 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.89 2021-04-26 02:42:09,005 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6020.686458 2021-04-26 02:42:10,772 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6020.686458 2021-04-26 02:42:12,664 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6020.686458 2021-04-26 02:42:13,588 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.99 % 2021-04-26 02:42:13,589 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.99 2021-04-26 02:42:14,421 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6020.686458 2021-04-26 02:42:16,180 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 6020.686458 2021-04-26 02:42:17,982 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7425.456267 2021-04-26 02:42:18,786 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 63.33 % 2021-04-26 02:42:18,786 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 63.33 2021-04-26 02:42:19,858 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13059.589364 2021-04-26 02:42:21,901 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13059.589364 2021-04-26 02:42:23,700 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13059.589364 2021-04-26 02:42:24,020 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 65.51 % 2021-04-26 02:42:24,022 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.51 2021-04-26 02:42:25,467 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13059.589364 2021-04-26 02:42:27,229 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20934.428932 2021-04-26 02:42:29,052 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32558.057226 2021-04-26 02:42:29,211 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 65.76 % 2021-04-26 02:42:29,211 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.76 2021-04-26 02:42:30,753 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32558.057226 2021-04-26 02:42:32,565 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32558.057226 2021-04-26 02:42:34,377 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32558.057226 2021-04-26 02:42:34,387 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.01 % 2021-04-26 02:42:34,387 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.01 2021-04-26 02:42:36,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32558.057226 2021-04-26 02:42:38,042 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32985.237232 2021-04-26 02:42:39,609 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.18 % 2021-04-26 02:42:39,611 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.18 2021-04-26 02:42:40,036 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33480.339474 2021-04-26 02:42:41,815 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33480.339474 2021-04-26 02:42:43,668 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33480.339474 2021-04-26 02:42:44,828 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.26 % 2021-04-26 02:42:44,828 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.26 2021-04-26 02:42:45,415 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33480.339474 2021-04-26 02:42:47,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 37896.620730 2021-04-26 02:42:49,003 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40910.169275 2021-04-26 02:42:49,996 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.41 % 2021-04-26 02:42:49,996 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.41 2021-04-26 02:42:50,858 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40910.169275 2021-04-26 02:42:52,673 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40910.169275 2021-04-26 02:42:54,581 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40910.169275 2021-04-26 02:42:55,193 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.59 % 2021-04-26 02:42:55,194 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.59 2021-04-26 02:42:56,489 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40243.835605 2021-04-26 02:42:58,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36350.112329 2021-04-26 02:43:00,296 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36350.112329 2021-04-26 02:43:00,371 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.78 % 2021-04-26 02:43:00,371 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.78 2021-04-26 02:43:02,049 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36350.112329 2021-04-26 02:43:03,905 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36350.112329 2021-04-26 02:43:05,556 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 66.87 % 2021-04-26 02:43:05,558 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.87 2021-04-26 02:43:05,664 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36350.112329 2021-04-26 02:43:07,559 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30661.938019 2021-04-26 02:43:09,440 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30661.938019 2021-04-26 02:43:10,723 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 67.04 % 2021-04-26 02:43:10,724 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.04 2021-04-26 02:43:11,476 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30661.938019 2021-04-26 02:43:13,450 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30661.938019 2021-04-26 02:43:15,315 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30661.938019 2021-04-26 02:43:15,936 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 67.27 % 2021-04-26 02:43:15,937 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.27 2021-04-26 02:43:17,236 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32102.659467 2021-04-26 02:43:19,045 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32082.681257 2021-04-26 02:43:20,740 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32082.681257 2021-04-26 02:43:21,122 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 67.40 % 2021-04-26 02:43:21,124 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.4 2021-04-26 02:43:22,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32082.681257 2021-04-26 02:43:24,138 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32082.681257 2021-04-26 02:43:25,838 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32082.681257 2021-04-26 02:43:26,267 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 67.63 % 2021-04-26 02:43:26,269 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.63 2021-04-26 02:43:27,533 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:29,240 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:30,944 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:31,434 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 67.90 % 2021-04-26 02:43:31,436 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.9 2021-04-26 02:43:32,568 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:34,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:35,871 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25083.090990 2021-04-26 02:43:36,602 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 68.14 % 2021-04-26 02:43:36,602 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 68.14 2021-04-26 02:43:37,562 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:39,266 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:40,948 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:41,746 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 68.36 % 2021-04-26 02:43:41,746 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-04-26 02:43:42,634 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:44,239 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:45,855 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 12987.987417 2021-04-26 02:43:46,898 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 68.78 % 2021-04-26 02:43:46,900 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 68.78 2021-04-26 02:43:47,520 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1998.782522 2021-04-26 02:43:49,154 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1998.782522 2021-04-26 02:43:50,971 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1998.782522 2021-04-26 02:43:52,032 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.16 % 2021-04-26 02:43:52,033 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.16 2021-04-26 02:43:52,842 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1998.782522 2021-04-26 02:43:54,711 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1998.782522 2021-04-26 02:43:56,601 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5406.614455 2021-04-26 02:43:57,204 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.23 % 2021-04-26 02:43:57,206 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.23 2021-04-26 02:43:58,372 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20685.081913 2021-04-26 02:44:00,161 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20685.081913 2021-04-26 02:44:01,938 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20685.081913 2021-04-26 02:44:02,372 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.52 % 2021-04-26 02:44:02,374 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.52 2021-04-26 02:44:03,588 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20685.081913 2021-04-26 02:44:05,338 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20685.081913 2021-04-26 02:44:06,994 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32326.996986 2021-04-26 02:44:07,542 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.67 % 2021-04-26 02:44:07,542 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.67 2021-04-26 02:44:08,756 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39868.339116 2021-04-26 02:44:10,535 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39868.339116 2021-04-26 02:44:12,313 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39868.339116 2021-04-26 02:44:12,760 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.86 % 2021-04-26 02:44:12,760 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.86 2021-04-26 02:44:14,062 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39868.339116 2021-04-26 02:44:15,806 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39868.339116 2021-04-26 02:44:17,490 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40751.622152 2021-04-26 02:44:18,019 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 69.96 % 2021-04-26 02:44:18,020 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.96 2021-04-26 02:44:19,290 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40751.622152 2021-04-26 02:44:21,075 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40751.622152 2021-04-26 02:44:22,857 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40751.622152 2021-04-26 02:44:23,196 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 70.11 % 2021-04-26 02:44:23,196 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.11 2021-04-26 02:44:24,707 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40751.622152 2021-04-26 02:44:26,463 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41157.751782 2021-04-26 02:44:28,211 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42551.019347 2021-04-26 02:44:28,375 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 70.35 % 2021-04-26 02:44:28,375 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.35 2021-04-26 02:44:30,005 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42551.019347 2021-04-26 02:44:31,884 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42551.019347 2021-04-26 02:44:33,538 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42551.019347 2021-04-26 02:44:33,553 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 70.61 % 2021-04-26 02:44:33,555 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.61 2021-04-26 02:44:35,338 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42551.019347 2021-04-26 02:44:37,118 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33560.359073 2021-04-26 02:44:38,710 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 70.68 % 2021-04-26 02:44:38,710 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.68 2021-04-26 02:44:38,880 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27687.570781 2021-04-26 02:44:40,684 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27687.570781 2021-04-26 02:44:42,405 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27687.570781 2021-04-26 02:44:43,881 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 70.89 % 2021-04-26 02:44:43,881 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.89 2021-04-26 02:44:44,148 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27687.570781 2021-04-26 02:44:45,880 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27687.570781 2021-04-26 02:44:47,625 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:49,028 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 71.15 % 2021-04-26 02:44:49,029 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.15 2021-04-26 02:44:49,351 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:51,157 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:52,871 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:54,187 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 71.24 % 2021-04-26 02:44:54,187 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.24 2021-04-26 02:44:54,569 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:56,269 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35787.600515 2021-04-26 02:44:58,042 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29449.217794 2021-04-26 02:44:59,336 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 71.48 % 2021-04-26 02:44:59,338 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.48 2021-04-26 02:44:59,719 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29449.217794 2021-04-26 02:45:01,427 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29449.217794 2021-04-26 02:45:03,124 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29449.217794 2021-04-26 02:45:04,486 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 71.77 % 2021-04-26 02:45:04,487 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.77 2021-04-26 02:45:04,775 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 29449.217794 2021-04-26 02:45:06,477 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 25987.900919 2021-04-26 02:45:08,114 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:09,634 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 71.98 % 2021-04-26 02:45:09,635 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.98 2021-04-26 02:45:09,733 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:11,361 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:12,994 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:14,651 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:14,767 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.13 % 2021-04-26 02:45:14,769 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.13 2021-04-26 02:45:16,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11934.460136 2021-04-26 02:45:17,894 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 5070.982688 2021-04-26 02:45:19,732 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 865.859911 2021-04-26 02:45:19,917 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.40 % 2021-04-26 02:45:19,918 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.4 2021-04-26 02:45:21,532 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 865.859911 2021-04-26 02:45:23,303 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 865.859911 2021-04-26 02:45:25,084 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.70 % 2021-04-26 02:45:25,085 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.7 2021-04-26 02:45:25,154 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 865.859911 2021-04-26 02:45:26,977 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 7938.345635 2021-04-26 02:45:28,744 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39361.851457 2021-04-26 02:45:30,256 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.87 % 2021-04-26 02:45:30,256 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.87 2021-04-26 02:45:30,599 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39361.851457 2021-04-26 02:45:32,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39361.851457 2021-04-26 02:45:34,174 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39361.851457 2021-04-26 02:45:35,453 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.92 % 2021-04-26 02:45:35,454 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.92 2021-04-26 02:45:35,951 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39361.851457 2021-04-26 02:45:37,755 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50990.407625 2021-04-26 02:45:39,575 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50990.407625 2021-04-26 02:45:40,622 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.97 % 2021-04-26 02:45:40,624 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.97 2021-04-26 02:45:41,483 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50990.407625 2021-04-26 02:45:43,243 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50990.407625 2021-04-26 02:45:45,040 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50990.407625 2021-04-26 02:45:45,786 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 72.99 % 2021-04-26 02:45:45,786 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.99 2021-04-26 02:45:46,795 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51006.107053 2021-04-26 02:45:48,533 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50363.994227 2021-04-26 02:45:50,213 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50363.994227 2021-04-26 02:45:50,963 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 73.00 % 2021-04-26 02:45:50,963 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 73.0 2021-04-26 02:45:51,931 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50363.994227 2021-04-26 02:45:53,648 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50363.994227 2021-04-26 02:45:55,411 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50363.994227 2021-04-26 02:45:56,144 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 73.03 % 2021-04-26 02:45:56,144 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 73.03 2021-04-26 02:45:57,190 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44185.377975 2021-04-26 02:45:58,911 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40096.757056 2021-04-26 02:46:00,552 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40096.757056 2021-04-26 02:46:01,322 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 73.08 % 2021-04-26 02:46:01,322 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 73.08 2021-04-26 02:46:02,223 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40096.757056 2021-04-26 02:46:03,881 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40096.757056 2021-04-26 02:46:05,517 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 40096.757056 2021-04-26 02:46:06,464 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 73.25 % 2021-04-26 02:46:06,466 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 73.25 2021-04-26 02:46:07,187 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 26910.767530 2021-04-26 02:46:08,806 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17707.730908 2021-04-26 02:46:10,451 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17707.730908 2021-04-26 02:46:11,634 | test | DEBUG | pool-2-thread-5 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 73.28 % 2021-04-26 02:46:11,634 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 73.28 2021-04-26 02:46:12,095 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17707.730908 2021-04-26 02:46:13,757 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17707.730908 2021-04-26 02:46:15,355 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 17707.730908 2021-04-26 02:46:17,061 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15451.189655 2021-04-26 02:46:18,647 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14307.455266 2021-04-26 02:46:20,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14307.455266 2021-04-26 02:46:21,789 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-26 02:46:21,848 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14307.455266 2021-04-26 02:46:23,446 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14307.455266 2021-04-26 02:46:25,049 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14307.455266 2021-04-26 02:46:26,619 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13888.528155 2021-04-26 02:46:28,184 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:29,730 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:31,280 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:32,821 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:34,365 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:35,913 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 13473.866203 2021-04-26 02:46:37,466 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:38,997 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:40,710 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:42,401 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:44,138 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:45,874 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2560.456728 2021-04-26 02:46:47,536 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27057.818853 2021-04-26 02:46:49,308 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27057.818853 2021-04-26 02:46:51,066 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27057.818853 2021-04-26 02:46:52,019 | test | ERROR | pool-2-thread-5 | [task:check:409] Node 172.23.121.81:8091 was not cleaned after removing from cluster 2021-04-26 02:46:52,020 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 444.743999958 sec 2021-04-26 02:46:52,022 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-26 02:46:52,367 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:46:52,368 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:46:52,368 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:46:52,368 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:46:52,368 | test | DEBUG | MainThread | [cluster:scenario_rebalance_out:87] Removing node with services: eventing,fts 2021-04-26 02:46:52,703 | test | DEBUG | pool-2-thread-6 | [task:start_rebalance:326] Removing node 172.23.136.110:8091 from cluster 2021-04-26 02:46:52,704 | test | DEBUG | pool-2-thread-6 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': u'ns_1@172.23.136.110', 'user': 'Administrator'} 2021-04-26 02:46:52,782 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27057.818853 2021-04-26 02:46:52,796 | test | DEBUG | pool-2-thread-6 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:46:52,798 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 17.6043133693 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 2.63076756414 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 36.5121957317 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 1.49752495875 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 12.8772853786 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 12.8258333333 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 3.2941117648 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 13.6955706488 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 1.72978999567 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 0.286669055575 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 4.43064872297 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 2.05748285431 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 39.4535091082 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 3.41 | --- OUT ---> | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 9.53166666667 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-26 02:46:52,835 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 0.00 % 2021-04-26 02:46:52,835 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-26 02:46:54,700 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 27057.818853 2021-04-26 02:46:56,608 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31894.317553 2021-04-26 02:46:58,226 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 33.87 % 2021-04-26 02:46:58,226 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 33.87 2021-04-26 02:46:58,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50073.236381 2021-04-26 02:47:00,196 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50073.236381 2021-04-26 02:47:01,913 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50073.236381 2021-04-26 02:47:03,405 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 34.76 % 2021-04-26 02:47:03,407 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 34.76 2021-04-26 02:47:03,637 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50073.236381 2021-04-26 02:47:05,331 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50073.236381 2021-04-26 02:47:07,082 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49853.840365 2021-04-26 02:47:08,581 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 35.23 % 2021-04-26 02:47:08,582 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 35.23 2021-04-26 02:47:08,736 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50252.771953 2021-04-26 02:47:10,369 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50252.771953 2021-04-26 02:47:12,003 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50252.771953 2021-04-26 02:47:13,673 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50252.771953 2021-04-26 02:47:13,742 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 35.26 % 2021-04-26 02:47:13,742 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 35.26 2021-04-26 02:47:15,301 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50252.771953 2021-04-26 02:47:16,967 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42368.544950 2021-04-26 02:47:18,581 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35858.811331 2021-04-26 02:47:18,904 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 35.69 % 2021-04-26 02:47:18,905 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 35.69 2021-04-26 02:47:20,171 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35858.811331 2021-04-26 02:47:21,785 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35858.811331 2021-04-26 02:47:23,348 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35858.811331 2021-04-26 02:47:24,036 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 35.73 % 2021-04-26 02:47:24,038 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 35.73 2021-04-26 02:47:24,934 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35858.811331 2021-04-26 02:47:26,476 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28478.114545 2021-04-26 02:47:28,040 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:29,207 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.02 % 2021-04-26 02:47:29,209 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.02 2021-04-26 02:47:29,595 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:31,211 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:32,752 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:34,289 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:34,338 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.10 % 2021-04-26 02:47:34,338 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.1 2021-04-26 02:47:35,848 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 18001.407727 2021-04-26 02:47:37,401 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:38,928 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:39,448 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.13 % 2021-04-26 02:47:39,448 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.13 2021-04-26 02:47:40,743 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:42,496 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:44,263 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:44,607 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.36 % 2021-04-26 02:47:44,608 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.36 2021-04-26 02:47:46,075 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 2240.816402 2021-04-26 02:47:47,848 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20522.070492 2021-04-26 02:47:49,605 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33022.080489 2021-04-26 02:47:49,793 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.41 % 2021-04-26 02:47:49,795 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.41 2021-04-26 02:47:51,427 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33022.080489 2021-04-26 02:47:53,171 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33022.080489 2021-04-26 02:47:54,940 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.43 % 2021-04-26 02:47:54,940 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-04-26 02:47:54,941 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33022.080489 2021-04-26 02:47:56,776 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 46563.420558 2021-04-26 02:47:58,660 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 55697.791310 2021-04-26 02:48:00,111 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.67 % 2021-04-26 02:48:00,111 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.67 2021-04-26 02:48:00,457 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 55697.791310 2021-04-26 02:48:02,322 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 55697.791310 2021-04-26 02:48:04,111 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 55697.791310 2021-04-26 02:48:05,273 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 36.67 % 2021-04-26 02:48:05,273 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.67 2021-04-26 02:48:05,913 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 55697.791310 2021-04-26 02:48:07,688 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 56045.234497 2021-04-26 02:48:09,424 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 56045.234497 2021-04-26 02:48:10,451 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 40.00 % 2021-04-26 02:48:10,453 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-04-26 02:48:11,237 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 56045.234497 2021-04-26 02:48:12,950 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 56045.234497 2021-04-26 02:48:14,904 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 56045.234497 2021-04-26 02:48:15,621 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 40.00 % 2021-04-26 02:48:15,622 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-04-26 02:48:16,588 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 50529.731755 2021-04-26 02:48:18,368 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41979.418997 2021-04-26 02:48:20,035 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41979.418997 2021-04-26 02:48:20,782 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 40.00 % 2021-04-26 02:48:20,783 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-04-26 02:48:21,733 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41979.418997 2021-04-26 02:48:23,368 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41979.418997 2021-04-26 02:48:24,967 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 41979.418997 2021-04-26 02:48:25,903 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 40.00 % 2021-04-26 02:48:25,904 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-04-26 02:48:26,637 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34920.581541 2021-04-26 02:48:28,210 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:29,823 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:31,023 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:31,023 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:31,430 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:33,042 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:34,624 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:36,153 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:36,154 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:36,198 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 24627.424568 2021-04-26 02:48:37,769 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20954.584208 2021-04-26 02:48:39,413 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15493.836759 2021-04-26 02:48:41,069 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15493.836759 2021-04-26 02:48:41,278 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:41,278 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:42,599 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15493.836759 2021-04-26 02:48:44,153 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15493.836759 2021-04-26 02:48:45,742 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15493.836759 2021-04-26 02:48:46,401 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:46,401 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:47,289 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11908.373456 2021-04-26 02:48:48,826 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10924.455766 2021-04-26 02:48:50,378 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10924.455766 2021-04-26 02:48:51,526 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:51,526 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:51,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10924.455766 2021-04-26 02:48:53,460 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10924.455766 2021-04-26 02:48:55,000 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10924.455766 2021-04-26 02:48:56,667 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:48:56,668 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:48:56,897 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8815.339425 2021-04-26 02:48:58,707 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1263.225881 2021-04-26 02:49:00,489 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1263.225881 2021-04-26 02:49:01,878 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:01,880 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:02,230 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1263.225881 2021-04-26 02:49:03,979 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1263.225881 2021-04-26 02:49:05,756 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 1263.225881 2021-04-26 02:49:07,051 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:07,051 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:07,595 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33031.723415 2021-04-26 02:49:09,344 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54175.356639 2021-04-26 02:49:11,121 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54175.356639 2021-04-26 02:49:12,214 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:12,214 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:12,900 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54175.356639 2021-04-26 02:49:14,727 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54175.356639 2021-04-26 02:49:16,487 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53863.559725 2021-04-26 02:49:17,384 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:17,384 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:18,200 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 52000.764301 2021-04-26 02:49:20,000 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 52000.764301 2021-04-26 02:49:21,757 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 52000.764301 2021-04-26 02:49:22,539 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:22,539 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:23,471 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 52000.764301 2021-04-26 02:49:25,213 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 52000.764301 2021-04-26 02:49:26,961 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51050.513923 2021-04-26 02:49:27,686 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:27,687 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:28,658 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 47006.504518 2021-04-26 02:49:30,450 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 47006.504518 2021-04-26 02:49:32,220 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 47006.504518 2021-04-26 02:49:32,841 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:32,842 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:33,832 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 47006.504518 2021-04-26 02:49:35,433 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 47006.504518 2021-04-26 02:49:37,026 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44520.658429 2021-04-26 02:49:37,983 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:37,984 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:38,602 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34412.946238 2021-04-26 02:49:40,269 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34412.946238 2021-04-26 02:49:41,871 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34412.946238 2021-04-26 02:49:43,109 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:43,111 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:43,493 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34412.946238 2021-04-26 02:49:45,078 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 34412.946238 2021-04-26 02:49:46,670 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 30637.770634 2021-04-26 02:49:48,239 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:48,239 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:48,247 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:49,822 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:51,447 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:53,000 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:53,367 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:53,367 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:54,552 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:56,128 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 15887.905523 2021-04-26 02:49:57,703 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 14465.476417 2021-04-26 02:49:58,490 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:49:58,490 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:49:59,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8315.208574 2021-04-26 02:50:00,796 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8315.208574 2021-04-26 02:50:02,348 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8315.208574 2021-04-26 02:50:03,607 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:03,608 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:04,108 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8315.208574 2021-04-26 02:50:05,862 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 8315.208574 2021-04-26 02:50:07,688 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 10195.531863 2021-04-26 02:50:08,779 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:08,780 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:09,482 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11503.950141 2021-04-26 02:50:11,247 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11503.950141 2021-04-26 02:50:13,052 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11503.950141 2021-04-26 02:50:13,936 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:13,936 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:14,881 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 11503.950141 2021-04-26 02:50:16,619 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 20454.558479 2021-04-26 02:50:18,382 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54696.913001 2021-04-26 02:50:19,101 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:19,102 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:20,217 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54696.913001 2021-04-26 02:50:21,983 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54696.913001 2021-04-26 02:50:23,684 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54696.913001 2021-04-26 02:50:24,269 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:24,270 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:25,437 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54696.913001 2021-04-26 02:50:27,190 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 54118.703045 2021-04-26 02:50:28,947 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53438.837498 2021-04-26 02:50:29,450 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:29,451 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:30,780 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53438.837498 2021-04-26 02:50:32,588 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53438.837498 2021-04-26 02:50:34,631 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:34,631 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:34,694 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53438.837498 2021-04-26 02:50:36,793 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 53438.837498 2021-04-26 02:50:38,828 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49252.450530 2021-04-26 02:50:39,844 | test | DEBUG | pool-2-thread-6 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:39,845 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:40,957 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49252.450530 2021-04-26 02:50:40,969 | test | INFO | pool-2-thread-6 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.120.100: 2021-04-26 02:50:40,969 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.115', u'tstamp': 1619430616404L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:50:16.404Z', u'text': u"The time on node 'ns_1@172.23.136.115' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,969 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.112', u'tstamp': 1619430600230L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:50:00.230Z', u'text': u"The time on node 'ns_1@172.23.136.112' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,970 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.114', u'tstamp': 1619430599342L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:49:59.342Z', u'text': u"The time on node 'ns_1@172.23.136.114' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,970 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.120.117', u'tstamp': 1619430588496L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:49:48.496Z', u'text': u"The time on node 'ns_1@172.23.120.117' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,970 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619430580012L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:49:40.012Z', u'text': u"The time on node 'ns_1@172.23.136.113' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,970 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.112', u'tstamp': 1619430539504L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:48:59.504Z', u'text': u"The time on node 'ns_1@172.23.136.112' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,970 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.114', u'tstamp': 1619430538498L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:48:58.498Z', u'text': u"The time on node 'ns_1@172.23.136.114' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,971 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.120.117', u'tstamp': 1619430528344L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:48:48.344Z', u'text': u"The time on node 'ns_1@172.23.120.117' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,971 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619430519260L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:48:39.260Z', u'text': u"The time on node 'ns_1@172.23.136.113' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,973 | test | ERROR | pool-2-thread-6 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.120.117', u'tstamp': 1619430468158L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:47:48.158Z', u'text': u"The time on node 'ns_1@172.23.120.117' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:50:40,973 | test | ERROR | pool-2-thread-6 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-04-26 02:50:40,973 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-26 02:50:41,552 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:50:41,552 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:50:41,553 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:50:41,553 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:50:42,059 | test | DEBUG | pool-2-thread-7 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': '', 'user': 'Administrator'} 2021-04-26 02:50:42,088 | test | DEBUG | pool-2-thread-7 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:50:42,092 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 88.998241652 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 48.0083001383 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 12.4775 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 0.314161430643 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 35.2221759066 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 31.3658333333 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 11.6016666667 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 23.882101965 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 5.69240512658 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 0.416677083594 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 12.2125 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 0.39 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 87.6291666667 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 2.34418620155 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 26.7823362251 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-26 02:50:42,128 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:42,130 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:43,107 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49252.450530 2021-04-26 02:50:45,216 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49252.450530 2021-04-26 02:50:47,142 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 49653.191100 2021-04-26 02:50:47,181 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:47,181 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:49,029 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51106.197966 2021-04-26 02:50:51,026 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51106.197966 2021-04-26 02:50:52,375 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:52,375 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:52,894 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51106.197966 2021-04-26 02:50:54,812 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51106.197966 2021-04-26 02:50:56,861 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 51106.197966 2021-04-26 02:50:57,571 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:50:57,572 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:50:58,898 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35116.861231 2021-04-26 02:51:00,941 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35116.861231 2021-04-26 02:51:02,796 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:02,798 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:03,017 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35116.861231 2021-04-26 02:51:05,042 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35116.861231 2021-04-26 02:51:07,134 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36719.511384 2021-04-26 02:51:08,042 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:08,043 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:09,121 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44136.431324 2021-04-26 02:51:11,134 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44136.431324 2021-04-26 02:51:13,112 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44136.431324 2021-04-26 02:51:13,256 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:13,257 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:15,073 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 44136.431324 2021-04-26 02:51:17,095 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42835.761391 2021-04-26 02:51:18,483 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:18,483 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:19,148 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36839.673002 2021-04-26 02:51:21,239 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36839.673002 2021-04-26 02:51:23,305 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36839.673002 2021-04-26 02:51:23,707 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:23,707 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:25,395 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36839.673002 2021-04-26 02:51:27,506 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 37088.729786 2021-04-26 02:51:28,924 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:28,924 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:29,483 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38039.829103 2021-04-26 02:51:31,469 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38039.829103 2021-04-26 02:51:33,466 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38039.829103 2021-04-26 02:51:34,145 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:34,147 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:35,428 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 38039.829103 2021-04-26 02:51:37,392 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39239.315887 2021-04-26 02:51:39,355 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:39,355 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:39,426 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39227.113020 2021-04-26 02:51:41,493 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39227.113020 2021-04-26 02:51:43,588 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39227.113020 2021-04-26 02:51:44,572 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:44,573 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:45,678 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39227.113020 2021-04-26 02:51:47,723 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35282.760421 2021-04-26 02:51:49,785 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:49,786 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:49,835 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35113.297668 2021-04-26 02:51:51,917 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35113.297668 2021-04-26 02:51:53,974 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35113.297668 2021-04-26 02:51:55,012 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:51:55,013 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:51:56,005 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 35113.297668 2021-04-26 02:51:57,970 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39722.529655 2021-04-26 02:51:59,967 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39722.529655 2021-04-26 02:52:00,213 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:00,213 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:01,926 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39722.529655 2021-04-26 02:52:04,023 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39722.529655 2021-04-26 02:52:05,461 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:05,463 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:06,124 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39722.529655 2021-04-26 02:52:08,200 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 36654.506931 2021-04-26 02:52:10,234 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32410.281530 2021-04-26 02:52:10,703 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:10,703 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:12,237 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32410.281530 2021-04-26 02:52:14,362 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32410.281530 2021-04-26 02:52:15,921 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:15,921 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:16,322 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32410.281530 2021-04-26 02:52:18,246 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 37025.638403 2021-04-26 02:52:20,219 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42848.165435 2021-04-26 02:52:21,132 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:21,132 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:22,196 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42848.165435 2021-04-26 02:52:24,151 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42848.165435 2021-04-26 02:52:26,082 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 42848.165435 2021-04-26 02:52:26,335 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:26,335 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:28,006 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 37091.198366 2021-04-26 02:52:29,957 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28450.628883 2021-04-26 02:52:31,529 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:31,530 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:31,928 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28450.628883 2021-04-26 02:52:33,930 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28450.628883 2021-04-26 02:52:35,996 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 28450.628883 2021-04-26 02:52:36,746 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:36,747 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:38,042 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 22437.323217 2021-04-26 02:52:40,035 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 19366.475284 2021-04-26 02:52:41,957 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:41,957 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:42,155 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 19366.475284 2021-04-26 02:52:44,296 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 19366.475284 2021-04-26 02:52:46,483 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 19366.475284 2021-04-26 02:52:47,188 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:47,188 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:48,641 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31883.955439 2021-04-26 02:52:50,750 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31883.955439 2021-04-26 02:52:52,430 | test | DEBUG | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:52,430 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:52,838 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31883.955439 2021-04-26 02:52:53,361 | test | INFO | pool-2-thread-7 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.120.100: 2021-04-26 02:52:53,362 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619430760488L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:52:40.488Z', u'text': u"The time on node 'ns_1@172.23.136.113' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,362 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619430752548L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:52:32.548Z', u'text': u"The time on node 'ns_1@172.23.136.105' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,364 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619430746042L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:52:26.042Z', u'text': u"The time on node 'ns_1@172.23.138.127' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,364 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.115', u'tstamp': 1619430737402L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:52:17.402Z', u'text': u"The time on node 'ns_1@172.23.136.115' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,364 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.112', u'tstamp': 1619430720562L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:52:00.562Z', u'text': u"The time on node 'ns_1@172.23.136.112' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,365 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.114', u'tstamp': 1619430719943L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:51:59.943Z', u'text': u"The time on node 'ns_1@172.23.136.114' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,365 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.120.117', u'tstamp': 1619430708109L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:51:48.109Z', u'text': u"The time on node 'ns_1@172.23.120.117' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,365 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.113', u'tstamp': 1619430700368L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:51:40.368Z', u'text': u"The time on node 'ns_1@172.23.136.113' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,365 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619430685363L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:51:25.363Z', u'text': u"The time on node 'ns_1@172.23.138.127' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,365 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.136.115', u'tstamp': 1619430677010L, u'shortText': u'message', u'serverTime': u'2021-04-26T02:51:17.010Z', u'text': u"The time on node 'ns_1@172.23.136.115' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized."} 2021-04-26 02:52:53,367 | test | ERROR | pool-2-thread-7 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-04-26 02:52:53,367 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-26 02:52:53,971 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.120.113 2021-04-26 02:52:53,973 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:59] Node service map: {u'backup': [ip:172.23.120.144 port:8091 ssh_username:Administrator], u'kv': [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator], u'index,n1ql': [ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator], u'eventing,fts': [ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator], u'cbas': [ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator]} 2021-04-26 02:52:53,973 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:60] Nodes in cluster: [ip:172.23.120.100 port:8091 ssh_username:Administrator, ip:172.23.120.113 port:8091 ssh_username:Administrator, ip:172.23.120.117 port:8091 ssh_username:Administrator, ip:172.23.120.144 port:8091 ssh_username:Administrator, ip:172.23.121.81 port:8091 ssh_username:Administrator, ip:172.23.136.105 port:8091 ssh_username:Administrator, ip:172.23.136.106 port:8091 ssh_username:Administrator, ip:172.23.136.107 port:8091 ssh_username:Administrator, ip:172.23.136.108 port:8091 ssh_username:Administrator, ip:172.23.136.110 port:8091 ssh_username:Administrator, ip:172.23.136.112 port:8091 ssh_username:Administrator, ip:172.23.136.113 port:8091 ssh_username:Administrator, ip:172.23.136.114 port:8091 ssh_username:Administrator, ip:172.23.136.115 port:8091 ssh_username:Administrator, ip:172.23.138.127 port:8091 ssh_username:Administrator] 2021-04-26 02:52:53,973 | test | DEBUG | MainThread | [cluster:update_nodes_in_cluster:61] Spare nodes: [] 2021-04-26 02:52:54,459 | test | DEBUG | pool-2-thread-8 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.120.100,ns_1@172.23.136.114,ns_1@172.23.136.106,ns_1@172.23.136.107,ns_1@172.23.120.113,ns_1@172.23.120.117,ns_1@172.23.138.127,ns_1@172.23.121.81,ns_1@172.23.120.144,ns_1@172.23.136.108,ns_1@172.23.136.112,ns_1@172.23.136.115,ns_1@172.23.136.113,ns_1@172.23.136.110,ns_1@172.23.136.105', 'ejectedNodes': '', 'user': 'Administrator'} 2021-04-26 02:52:54,479 | test | DEBUG | pool-2-thread-8 | [rest_client:rebalance:1344] Rebalance operation started 2021-04-26 02:52:54,480 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.120.100 | kv | 7.0.0-5014-enterprise | 97.724090803 | Cluster node | | 172.23.136.114 | index, n1ql | 7.0.0-5014-enterprise | 0.859159507004 | Cluster node | | 172.23.136.106 | index, n1ql | 7.0.0-5014-enterprise | 0.698922494488 | Cluster node | | 172.23.136.107 | cbas | 7.0.0-5014-enterprise | 14.2421414047 | Cluster node | | 172.23.120.113 | kv | 7.0.0-5014-enterprise | 96.8233068609 | Cluster node | | 172.23.120.117 | kv | 7.0.0-5014-enterprise | 98.2941524513 | Cluster node | | 172.23.138.127 | eventing, fts | 7.0.0-5014-enterprise | 23.3058333333 | Cluster node | | 172.23.121.81 | index, n1ql | 7.0.0-5014-enterprise | 13.7501145843 | Cluster node | | 172.23.120.144 | backup | 7.0.0-5014-enterprise | 5.25995616703 | Cluster node | | 172.23.136.108 | cbas | 7.0.0-5014-enterprise | 14.3621409465 | Cluster node | | 172.23.136.112 | eventing, fts | 7.0.0-5014-enterprise | 32.7472271064 | Cluster node | | 172.23.136.115 | cbas | 7.0.0-5014-enterprise | 8.92075899368 | Cluster node | | 172.23.136.113 | kv | 7.0.0-5014-enterprise | 95.5216293469 | Cluster node | | 172.23.136.110 | eventing, fts | 7.0.0-5014-enterprise | 20.0896689084 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5014-enterprise | 95.8208681594 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-26 02:52:54,542 | test | DEBUG | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:54,543 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:52:55,184 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 31883.955439 2021-04-26 02:52:57,256 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 32090.055439 2021-04-26 02:52:59,434 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33862.940686 2021-04-26 02:52:59,611 | test | DEBUG | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:52:59,611 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:53:01,844 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33862.940686 2021-04-26 02:53:04,096 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33862.940686 2021-04-26 02:53:04,819 | test | DEBUG | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:53:04,821 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:53:06,500 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 33862.940686 2021-04-26 02:53:08,832 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39384.744805 2021-04-26 02:53:10,072 | test | DEBUG | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:53:10,072 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:53:11,039 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39384.744805 2021-04-26 02:53:13,355 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39384.744805 2021-04-26 02:53:15,306 | test | DEBUG | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1515] Rebalance percentage: 41.81 % 2021-04-26 02:53:15,308 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 41.81 2021-04-26 02:53:15,782 | test | DEBUG | pool-2-thread-1 | [task:record_bucket_ops:3457] Ops rate for 'travel-sample': 39384.744805 Build was aborted Aborted by Ashwin G Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion No emails were triggered. Finished: ABORTED