Skip to content titletitle Ashwin G | log out Jenkins oel6-4node-rebalance_in_jython #1079 Back to Project Status Changes Console Output View as plain text Edit Build Information Delete Build Parameters Downstream build view Environment Variables Git Build Data No Tags Test Result Failure Cause Management Rebuild Delete log Previous Build FailedConsole Output Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s71802-32g-12c (P0 jython_slave PY3P0 xattr-subdoc-andrei P20) in workspace /data/workspace/oel6-4node-rebalance_in_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/oel6-4node-rebalance_in_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/mad-hatter^{commit} # timeout=10 Checking out Revision b393af72346555687b2f51249f1be3d1f3f594cd (origin/mad-hatter) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f b393af72346555687b2f51249f1be3d1f3f594cd > /usr/bin/git rev-list 9c7b4d822613e1625837c2c5d7170547a581bb21 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #1079 jenkins-oel6-4node-rebalance_in_jython-1079 # timeout=10 No emails were triggered. [oel6-4node-rebalance_in_jython] $ /bin/bash -x /tmp/jenkins5628087441877662937.sh + jython_path=/opt/jython/bin/jython + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + echo ' [global] username:root password:couchbase #username:Administrator #password:Membase123 port:8091 n1ql_port:8093 index_port:9102 # data_path:/data/dta # index_path:/data/idx # cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.105.155 port:8091 services:kv,n1ql,index,fts [_2] ip:172.23.105.159 port:8091 services:kv [_3] ip:172.23.105.205 port:8091 services:kv [_4] ip:172.23.105.206 port:8091 services:kv [_5] ip:172.23.105.229 port:8091 services:kv [_6] ip:172.23.105.230 port:8091 services:kv [membase] rest_username:Administrator rest_password:password ' + echo 'rebalance_new.swaprebalancetests.SwapRebalanceBasicTests: do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True #rebalance_new.rebalance_out.RebalanceOutTests: # rebalance_out_after_ops,doc_size=512,transaction_timeout=150,upgrade_version=6.6.0-7883,rerun=False,doc_ops=update,nodes_out=1,transaction_commit=TRUE,atomicity=True,replicas=1,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,log_level=error,nodes_init=4,num_items=10000 #rebalance_new.rebalance_durability.RebalanceDurability.test_replica_update_with_durability_without_adding_removing_nodes,doc_size=512,defer=False,transaction_timeout=750,upgrade_version=6.6.0-7883,nodes_in=3,rerun=True,GROUP=P0,afterTimePeriod=250,atomicity=True,get-cbcollect-info=False,replicas=0,log_level=debug,nodes_init=1,num_items=25000,infra_log_level=critical,num_items=10000,doc_size=512,transaction_timeout=150,atomicity=True,GROUP=P0,afterTimePeriod=250,rerun=False ' + rm -rf 'logs/*' jars + '[' false == true ']' + '[' 'git pull http://review.couchbase.org/TAF refs/changes/96/125896/13' '!=' '' ']' + git pull http://review.couchbase.org/TAF refs/changes/96/125896/13 From http://review.couchbase.org/TAF * branch refs/changes/96/125896/13 -> FETCH_HEAD Updating b393af7..10038d1 Fast-forward lib/Jython_tasks/task.py | 40 +++++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 13 deletions(-) + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/5-centos-nodes-jython.ini -c /tmp/test_run.conf -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -m rest' + tcpdump -i eth0 -s 0 -w /tmp/test.pcap tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes Starting a Gradle Daemon, 65 busy Daemons could not be reused, use --status for details > Configure project : Running: /opt/jython/bin/jython -J-cp /data/workspace/oel6-4node-rebalance_in_jython/build/classes/java/main:/data/workspace/oel6-4node-rebalance_in_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0-SNAPSHOT/37d44178dc2fe3a55d60d290bb4d276c48dabdcb/couchbase-transactions-1.1.0-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.6-SNAPSHOT/c7c272d4725e35978992fde8b2b22067eecc89df/java-client-3.0.6-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.7-SNAPSHOT/112277b50e255af1ec1d5454320c889875935da3/core-io-2.0.7-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/5-centos-nodes-jython.ini -c /tmp/test_run.conf -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/oel6-4node-rebalance_in_jython/build/classes/java/main:/data/workspace/oel6-4node-rebalance_in_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0-SNAPSHOT/37d44178dc2fe3a55d60d290bb4d276c48dabdcb/couchbase-transactions-1.1.0-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.6-SNAPSHOT/c7c272d4725e35978992fde8b2b22067eecc89df/java-client-3.0.6-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.7-SNAPSHOT/112277b50e255af1ec1d5454320c889875935da3/core-io-2.0.7-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/5-centos-nodes-jython.ini -c /tmp/test_run.conf -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/oel6-4node-rebalance_in_jython/build/classes/java/main:/data/workspace/oel6-4node-rebalance_in_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0-SNAPSHOT/37d44178dc2fe3a55d60d290bb4d276c48dabdcb/couchbase-transactions-1.1.0-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.6-SNAPSHOT/c7c272d4725e35978992fde8b2b22067eecc89df/java-client-3.0.6-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.7-SNAPSHOT/112277b50e255af1ec1d5454320c889875935da3/core-io-2.0.7-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/5-centos-nodes-jython.ini -c /tmp/test_run.conf -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -m rest > Task :compileJava Note: /data/workspace/oel6-4node-rebalance_in_jython/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: /tmp/test_run.conf Prefix: rebalance_new.swaprebalancetests.SwapRebalanceBasicTests Global Test input params: {'cluster_name': '5-centos-nodes-jython', 'collect_pcaps': 'True', 'conf_file': '/tmp/test_run.conf', 'get-cbcollect-info': 'False', 'ini': '/tmp/5-centos-nodes-jython.ini', 'num_nodes': 6, 'rerun': 'False', 'spec': 'test_run'} Logs will be stored at /data/workspace/oel6-4node-rebalance_in_jython/logs/testrunner-20-Jul-17_07-57-44/test_1 ./testrunner -i /tmp/5-centos-nodes-jython.ini rerun=False,get-cbcollect-info=False,collect_pcaps=True -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True Test Input params: {'doc_size': '512', 'cluster_name': '5-centos-nodes-jython', 'atomicity': 'True', 'standard_buckets': '2', 'ini': '/tmp/5-centos-nodes-jython.ini', 'get-cbcollect-info': 'False', 'replicas': '2', 'transaction_timeout': '150', 'durability': 'PERSIST_TO_MAJORITY', 'swap-orchestrator': 'True', 'conf_file': '/tmp/test_run.conf', 'spec': 'test_run', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/oel6-4node-rebalance_in_jython/logs/testrunner-20-Jul-17_07-57-44/test_1', 'num-swap': '2', 'nodes_init': '4', 'num_items': '10000', 'collect_pcaps': 'True', 'case_number': 1} 2020-07-17 07:57:45,509 | root | WARNING | MainThread | [data_helper::31] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially do_test (rebalance_new.swaprebalancetests.SwapRebalanceBasicTests) ... 2020-07-17 07:58:32,918 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:481] ========= basetestcase setup started for test #1 do_test ========= 2020-07-17 07:58:34,115 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:481] ========= basetestcase setup started for test #1 do_test ========= 2020-07-17 07:58:35,180 | test | ERROR | MainThread | [rest_client:_http_request:737] GET http://172.23.105.155:8091/pools/default/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2020-07-17 07:59:06,341 | test | INFO | MainThread | [basetestcase:setUp:275] Initializing cluster 2020-07-17 07:59:06,674 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:512] RAM quota was defined less than 100 MB: 2020-07-17 07:59:09,755 | test | INFO | MainThread | [basetestcase:setUp:297] Couchbase Cluster: default, Nodes: 172.23.105.155, 172.23.105.159, 172.23.105.205, 172.23.105.206, 172.23.105.229, 172.23.105.230 initialized 2020-07-17 07:59:09,779 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:481] ========= basetestcase setup finished for test #1 do_test ========= 2020-07-17 07:59:10,342 | test | INFO | MainThread | [basetestcase:sleep:583] Reason: None. Sleep for 5 secs ... 2020-07-17 07:59:21,243 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------------------+--------------+ | Nodes | Services | Status | +----------------+----------------------+--------------+ | 172.23.105.155 | fts, index, kv, n1ql | Cluster node | | 172.23.105.159 | None | <--- IN --- | | 172.23.105.205 | None | <--- IN --- | | 172.23.105.206 | None | <--- IN --- | +----------------+----------------------+--------------+ 2020-07-17 07:59:21,269 | test | INFO | pool-2-thread-7 | [task:check:304] Rebalance - status: running, progress: 0 2020-07-17 07:59:36,339 | test | INFO | pool-2-thread-7 | [task:check:304] Rebalance - status: none, progress: 100 2020-07-17 07:59:36,369 | test | INFO | pool-2-thread-7 | [task:check:363] Rebalance completed with progress: 100% in 15.1279997826 sec 2020-07-17 07:59:52,253 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.155:11210 bucket-0 2020-07-17 07:59:52,506 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.159:11210 bucket-0 2020-07-17 07:59:52,736 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.205:11210 bucket-0 2020-07-17 07:59:52,960 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.206:11210 bucket-0 2020-07-17 07:59:53,319 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.155:11210 bucket-0 2020-07-17 07:59:53,611 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.159:11210 bucket-0 2020-07-17 07:59:53,818 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.205:11210 bucket-0 2020-07-17 07:59:54,007 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.206:11210 bucket-0 2020-07-17 07:59:55,803 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.155:11210 bucket-1 2020-07-17 07:59:55,930 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.159:11210 bucket-1 2020-07-17 07:59:56,053 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.205:11210 bucket-1 2020-07-17 07:59:56,176 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.206:11210 bucket-1 2020-07-17 07:59:56,378 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.155:11210 bucket-1 2020-07-17 07:59:56,503 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.159:11210 bucket-1 2020-07-17 07:59:56,632 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.205:11210 bucket-1 2020-07-17 07:59:56,946 | test | INFO | MainThread | [data_helper:direct_client:268] creating direct client 172.23.105.206:11210 bucket-1 2020-07-17 08:00:02,747 | test | INFO | pool-2-thread-10 | [task:call:3691] Transaction timeout: 150 2020-07-17 08:00:02,813 | test | INFO | pool-2-thread-10 | [task:call:3695] Transaction: com.couchbase.transactions.Transactions@2124c8f4 TransactionsStarted: Transactions successfully started, regular cleanup enabled=true, lost cleanup enabled=true 2020-07-17 08:00:02,911 | test | INFO | pool-2-thread-4 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,924 | test | INFO | pool-2-thread-5 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,924 | test | INFO | pool-2-thread-1 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,924 | test | INFO | pool-2-thread-3 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,924 | test | INFO | pool-2-thread-9 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,911 | test | INFO | pool-2-thread-6 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,926 | test | INFO | pool-2-thread-2 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:02,924 | test | INFO | pool-2-thread-7 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:43,661 | test | INFO | pool-2-thread-2 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:43,664 | test | INFO | pool-2-thread-2 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:43,736 | test | INFO | pool-2-thread-1 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:43,737 | test | INFO | pool-2-thread-1 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:43,769 | test | INFO | pool-2-thread-7 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:43,769 | test | INFO | pool-2-thread-7 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:43,831 | test | INFO | pool-2-thread-4 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:43,832 | test | INFO | pool-2-thread-4 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:43,871 | test | INFO | pool-2-thread-5 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:43,872 | test | INFO | pool-2-thread-5 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:44,015 | test | INFO | pool-2-thread-3 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:44,016 | test | INFO | pool-2-thread-3 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:44,118 | test | INFO | pool-2-thread-6 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:44,121 | test | INFO | pool-2-thread-6 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:44,250 | test | INFO | pool-2-thread-9 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:00:44,253 | test | INFO | pool-2-thread-9 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:00:45,000 | test | INFO | pool-2-thread-7 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,028 | test | INFO | pool-2-thread-1 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,078 | test | INFO | pool-2-thread-2 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,131 | test | INFO | pool-2-thread-4 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,141 | test | INFO | pool-2-thread-6 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,157 | test | INFO | pool-2-thread-9 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,184 | test | INFO | pool-2-thread-3 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,191 | test | INFO | pool-2-thread-5 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:00:45,923 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.105.205 | kv | 54.8051948052 | 4201840640 | 3431661568 | 0 / 3758092288 | 5175 / 10387 | 6.6.0-7883-enterprise | | 172.23.105.155 | fts, index, kv, n1ql | 63.8961038961 | 4201840640 | 3373981696 | 0 / 3758092288 | 5055 / 10132 | 6.6.0-7883-enterprise | | 172.23.105.206 | kv | 61.9537275064 | 4201840640 | 3430531072 | 0 / 3758092288 | 4789 / 9486 | 6.6.0-7883-enterprise | | 172.23.105.159 | kv | 57.8534031414 | 4201840640 | 3407785984 | 0 / 3758092288 | 4922 / 9935 | 6.6.0-7883-enterprise | +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2020-07-17 08:00:46,029 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ | bucket-0 | membase | 2 | none | 0 | 10342 | 2986344448 | 98332384 | 551673306 | | bucket-1 | membase | 2 | none | 0 | 9599 | 2986344448 | 92995920 | 467870372 | +----------+---------+----------+------------+-----+-------+------------+----------+-----------+ 2020-07-17 08:00:46,030 | test | INFO | MainThread | [rebalance_base:setUp:85] ==========Finished rebalance base setup======== 2020-07-17 08:00:46,032 | test | INFO | MainThread | [swaprebalancetests:setUp:21] === SwapRebalanceBase setup started for test #1 do_test === 2020-07-17 08:00:46,058 | test | INFO | MainThread | [swaprebalancetests:setUp:44] picking server : ip:172.23.105.155 port:8091 ssh_username:root as the master 2020-07-17 08:00:46,711 | test | INFO | MainThread | [swaprebalancetests:enable_diag_eval_on_non_local_hosts:90] Enabled diag/eval for non-local hosts from 172.23.105.155 2020-07-17 08:00:46,757 | test | INFO | MainThread | [swaprebalancetests:setUp:69] === SwapRebalanceBase setup done for test #1 do_test === 2020-07-17 08:00:48,404 | test | INFO | pool-2-thread-8 | [task:call:3691] Transaction timeout: 150 2020-07-17 08:00:48,408 | test | INFO | pool-2-thread-8 | [task:call:3695] Transaction: com.couchbase.transactions.Transactions@22d891c7 2020-07-17 08:00:48,454 | test | INFO | pool-2-thread-7 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,456 | test | INFO | pool-2-thread-3 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,454 | test | INFO | pool-2-thread-1 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,454 | test | INFO | pool-2-thread-4 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,454 | test | INFO | pool-2-thread-2 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,456 | test | INFO | pool-2-thread-6 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,456 | test | INFO | pool-2-thread-9 | [task:call:3819] Starting Atomicity load generation thread TransactionsStarted: Transactions successfully started, regular cleanup enabled=true, lost cleanup enabled=true 2020-07-17 08:00:48,469 | test | INFO | pool-2-thread-5 | [task:call:3819] Starting Atomicity load generation thread 2020-07-17 08:00:48,513 | test | INFO | MainThread | [swaprebalancetests:_common_test_body_swap_rebalance:130] Current nodes: [u'ns_1@172.23.105.205', u'ns_1@172.23.105.155', u'ns_1@172.23.105.206', u'ns_1@172.23.105.159'] 2020-07-17 08:00:49,282 | test | INFO | MainThread | [swaprebalancetests:_common_test_body_swap_rebalance:145] removing node ns_1@172.23.105.155 and rebalance afterwards 2020-07-17 08:00:49,282 | test | INFO | MainThread | [swaprebalancetests:_common_test_body_swap_rebalance:145] removing node ns_1@172.23.105.206 and rebalance afterwards 2020-07-17 08:00:54,132 | test | INFO | MainThread | [swaprebalancetests:_common_test_body_swap_rebalance:181] Swap Rebalance phase DurabilityImpossibleException seen 2020-07-17 08:00:58,535 | test | INFO | pool-2-thread-1 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:03,641 | test | INFO | pool-2-thread-4 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:17,582 | test | INFO | pool-2-thread-6 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:23,420 | test | INFO | pool-2-thread-7 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:25,250 | test | INFO | pool-2-thread-9 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:25,971 | test | INFO | pool-2-thread-3 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... DurabilityImpossibleException seen 2020-07-17 08:01:45,614 | test | INFO | pool-2-thread-2 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... 2020-07-17 08:01:48,190 | test | INFO | pool-2-thread-5 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:01:48,190 | test | INFO | pool-2-thread-5 | [task:call:3951] Starting Atomicity Verification thread TransactionCleanupEndRunEvent: start={bucket-0/_default/clientId=8a883,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=59800millis,ATRs={read=1024,errored=0,present=884},entries={expired=0,failed=0,total=2537},maxEntries=10} 2020-07-17 08:01:48,414 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 2020-07-17 08:01:48,615 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 DurabilityImpossibleException seen 2020-07-17 08:02:03,977 | test | INFO | pool-2-thread-1 | [task:transaction_load:4006] DurabilityImpossibleException seen so retrying... 2020-07-17 08:02:18,831 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 2020-07-17 08:02:19,023 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 2020-07-17 08:02:30,329 | test | INFO | pool-2-thread-4 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:30,331 | test | INFO | pool-2-thread-4 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:30,558 | test | WARNING | pool-2-thread-4 | [task:call:3974] Keys missing: test_docs-00013922,test_docs-00013912 2020-07-17 08:02:30,769 | test | WARNING | pool-2-thread-4 | [task:call:3974] Keys missing: test_docs-00013922,test_docs-00013912 2020-07-17 08:02:36,977 | test | INFO | pool-2-thread-6 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:36,980 | test | INFO | pool-2-thread-6 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:37,180 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015206,test_docs-00015217,test_docs-00015209,test_docs-00015327,test_docs-00015338 2020-07-17 08:02:37,374 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015206,test_docs-00015217,test_docs-00015209,test_docs-00015327,test_docs-00015338 2020-07-17 08:02:41,914 | test | INFO | pool-2-thread-7 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:41,915 | test | INFO | pool-2-thread-7 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:42,346 | test | INFO | pool-2-thread-7 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:02:42,823 | test | INFO | pool-2-thread-9 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:42,825 | test | INFO | pool-2-thread-9 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:43,262 | test | INFO | pool-2-thread-9 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:02:43,536 | test | INFO | pool-2-thread-3 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:43,536 | test | INFO | pool-2-thread-3 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:43,796 | test | WARNING | pool-2-thread-3 | [task:call:3974] Keys missing: test_docs-00017684 2020-07-17 08:02:44,003 | test | WARNING | pool-2-thread-3 | [task:call:3974] Keys missing: test_docs-00017684 2020-07-17 08:02:47,009 | test | INFO | pool-2-thread-2 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:02:47,012 | test | INFO | pool-2-thread-2 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:02:47,236 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012773,test_docs-00012888,test_docs-00012899 2020-07-17 08:02:47,460 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012773,test_docs-00012888,test_docs-00012899 TransactionCleanupEndRunEvent: start={bucket-0/_default/clientId=8a883,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=59714millis,ATRs={read=1024,errored=0,present=964},entries={expired=67,failed=0,total=3210},maxEntries=10,oldest=161684millis,id=_txn:atr-997-#1f08/4bdb6}} 2020-07-17 08:02:49,214 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 2020-07-17 08:02:49,401 | test | WARNING | pool-2-thread-5 | [task:call:3974] Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 2020-07-17 08:02:49,403 | test | INFO | pool-2-thread-5 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:03:00,967 | test | WARNING | pool-2-thread-4 | [task:call:3974] Keys missing: test_docs-00013922,test_docs-00013912 2020-07-17 08:03:01,163 | test | WARNING | pool-2-thread-4 | [task:call:3974] Keys missing: test_docs-00013922,test_docs-00013912 2020-07-17 08:03:04,579 | test | INFO | MainThread | [rest_client:monitorRebalance:1423] Rebalance done. Taken 130.351999998 seconds to complete 2020-07-17 08:03:04,581 | test | INFO | MainThread | [rest_client:monitorRebalance:1425] Sleep for 5 seconds after rebalance 2020-07-17 08:03:07,599 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015206,test_docs-00015217,test_docs-00015209,test_docs-00015327,test_docs-00015338 2020-07-17 08:03:07,809 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015206,test_docs-00015217,test_docs-00015209,test_docs-00015327,test_docs-00015338 2020-07-17 08:03:14,194 | test | WARNING | pool-2-thread-3 | [task:call:3974] Keys missing: test_docs-00017684 2020-07-17 08:03:14,378 | test | WARNING | pool-2-thread-3 | [task:call:3974] Keys missing: test_docs-00017684 2020-07-17 08:03:17,638 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012773,test_docs-00012888,test_docs-00012899 2020-07-17 08:03:17,802 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012773,test_docs-00012888,test_docs-00012899 2020-07-17 08:03:20,612 | test | INFO | pool-2-thread-1 | [task:call:3945] Atomicity Load generation thread completed 2020-07-17 08:03:20,614 | test | INFO | pool-2-thread-1 | [task:call:3951] Starting Atomicity Verification thread 2020-07-17 08:03:21,042 | test | INFO | pool-2-thread-1 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:03:31,585 | test | INFO | pool-2-thread-4 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:03:37,992 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015327,test_docs-00015338 2020-07-17 08:03:38,160 | test | WARNING | pool-2-thread-6 | [task:call:3974] Keys missing: test_docs-00015327,test_docs-00015338 2020-07-17 08:03:38,161 | test | INFO | pool-2-thread-6 | [task:call:3983] Completed Atomicity Verification thread 2020-07-17 08:03:44,766 | test | INFO | pool-2-thread-3 | [task:call:3983] Completed Atomicity Verification thread TransactionCleanupEndRunEvent: start={bucket-0/_default/clientId=8a883,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=59735millis,ATRs={read=1024,errored=0,present=997},entries={expired=1614,failed=0,total=3710},maxEntries=12,oldest=211202millis,id=_txn:atr-843-#b30/6253b}} 2020-07-17 08:03:47,986 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012888,test_docs-00012899 2020-07-17 08:03:48,165 | test | WARNING | pool-2-thread-2 | [task:call:3974] Keys missing: test_docs-00012888,test_docs-00012899 2020-07-17 08:03:48,167 | test | INFO | pool-2-thread-2 | [task:call:3983] Completed Atomicity Verification thread TransactionCleanupEndRunEvent: start={bucket-1/_default/clientId=8a883,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=59747millis,ATRs={read=1024,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={bucket-0/_default/clientId=8a883,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=59694millis,ATRs={read=1024,errored=0,present=999},entries={expired=319,failed=0,total=1492},maxEntries=7,oldest=211241millis,id=_txn:atr-103-#b0e/d9424}} ERROR 2020-07-17 08:04:48,808 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.105.229 | kv | 5.07614213198 | 4201840640 | 3313684480 | 0 / 3758092288 | 10253 / 20305 | 6.6.0-7883-enterprise | | 172.23.105.230 | kv | 5.03778337531 | 4201840640 | 3302985728 | 0 / 3758092288 | 10249 / 20688 | 6.6.0-7883-enterprise | | 172.23.105.205 | kv | 4.78589420655 | 4201840640 | 3328884736 | 0 / 3758092288 | 10251 / 20422 | 6.6.0-7883-enterprise | | 172.23.105.159 | kv | 4.54545454545 | 4201840640 | 3292954624 | 0 / 3758092288 | 10248 / 20587 | 6.6.0-7883-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2020-07-17 08:04:48,809 | test | INFO | MainThread | [basetestcase:check_coredump_exist:612] Initializing core dump check on all the nodes 2020-07-17 08:05:04,219 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.155 node pcap zip coped on client : True 2020-07-17 08:05:16,224 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.159 node pcap zip coped on client : True 2020-07-17 08:05:27,884 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.205 node pcap zip coped on client : True 2020-07-17 08:05:36,463 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.206 node pcap zip coped on client : True 2020-07-17 08:05:44,135 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.229 node pcap zip coped on client : True 2020-07-17 08:05:51,632 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:381] 172.23.105.230 node pcap zip coped on client : True 2020-07-17 08:05:51,723 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:440] Skipping get_trace !! 2020-07-17 08:06:19,997 | test | INFO | MainThread | [rest_client:monitorRebalance:1423] Rebalance done. Taken 25.0750000477 seconds to complete 2020-07-17 08:06:19,999 | test | INFO | MainThread | [rest_client:monitorRebalance:1425] Sleep for 5 seconds after rebalance ====================================================================== ERROR: do_test (rebalance_new.swaprebalancetests.SwapRebalanceBasicTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/swaprebalancetests.py", line 548, in do_test self._common_test_body_swap_rebalance(do_stop_start=False) File "pytests/rebalance_new/swaprebalancetests.py", line 253, in _common_test_body_swap_rebalance self.task_manager.get_task_result(task) File "lib/Jython_tasks/task_manager.py", line 35, in get_task_result return future.get() ExecutionException: java.util.concurrent.ExecutionException: Traceback (most recent call last): File "lib/Jython_tasks/task.py", line 3722, in call self.set_exception(exception_seen) File "lib/Jython_tasks/task.py", line 84, in set_exception raise BaseException(self.exception) BaseException: Keys missing: test_docs-00018944,test_docs-00018940,test_docs-00018938 ---------------------------------------------------------------------- Ran 1 test in 548.392s During the test, Remote Connections: 78, Disconnections: 78 SDK Connections: 32, Disconnections: 32 FAILED (errors=1) summary so far suite rebalance_new.swaprebalancetests.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test testrunner logs, diags and results are available under /data/workspace/oel6-4node-rebalance_in_jython/logs/testrunner-20-Jul-17_07-57-44/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 9m 41s 2 actionable tasks: 2 executed /tmp/jenkins5628087441877662937.sh: line 94: 2937 Killed tcpdump -i eth0 -s 0 -w /tmp/test.pcap + pkill tcpdump Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Failure - Any Sending email for trigger: Failure - Any An attempt to send an e-mail to empty list of recipients, ignored. Finished: FAILURE Help us localize this page Page generated: Jul 17, 2020 8:42:26 AM PDTREST APIJenkins ver. 2.108 SCM Sync status :