Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s72001 (P0 jython_slave PY3P0 xattr-subdoc-andrei) in workspace /data/workspace/temp_rebalance_even [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_rebalance_even # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision fb8c5b8822c92cf2362622175aaf0b8a41b09bda (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f fb8c5b8822c92cf2362622175aaf0b8a41b09bda > /usr/bin/git rev-list cd9e7b105438b515fa16246a397ff32a1950fd26 # timeout=10 [temp_rebalance_even] $ /bin/sh -xe /tmp/jenkins3834951439849859484.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 #data_path:/data/dta #index_path:/data/idx #cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [membase] rest_username:Administrator rest_password:password [cluster1] 1:_1 2:_2 [cluster2] 1:_3 2:_4 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 #6:_6 [_6] ip:172.23.98.196 port:8091 services:kv,index,n1ql [_2] ip:172.23.98.195 port:8091 services:kv,n1ql [_3] ip:172.23.121.10 port:8091 services:kv [_4] ip:172.23.104.186 port:8091 services:cbas [_5] ip:172.23.120.201 [_1] ip:172.23.120.206' + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 #6:_6 [_1] ip:172.23.98.196 [_2] ip:172.23.98.195 [_3] ip:172.23.121.10 [_4] ip:172.23.104.186 [_6] ip:172.23.120.201 [_5] ip:172.23.120.206' + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 70 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_rebalance_even/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 15s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_252 (Oracle Corporation 25.252-b09) OS: Linux 3.10.0-693.2.2.el7.x86_64 amd64 + git fetch http://review.couchbase.org/TAF refs/changes/15/147415/1 From http://review.couchbase.org/TAF * branch refs/changes/15/147415/1 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [master 0d79c93] temp fix to support xatttrs in collections tests Author: Balakumaran G 1 file changed, 6 insertions(+), 6 deletions(-) + guides/gradlew install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url=' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-4574,product=cb,parallel=,init_nodes=True,url= > Task :install 2021-03-02 00:08:05,733 - root - WARNING - URL: is invalid, will use version to locate build 2021-03-02 00:09:08,351 - root - INFO - Done with uninstall on 172.23.120.206. 2021-03-02 00:09:09,641 - root - INFO - Done with uninstall on 172.23.121.10. 2021-03-02 00:09:14,668 - root - INFO - Done with uninstall on 172.23.104.186. 2021-03-02 00:09:17,934 - root - INFO - Done with uninstall on 172.23.98.196. 2021-03-02 00:09:17,960 - root - INFO - Done with uninstall on 172.23.98.195. 2021-03-02 00:13:59,079 - root - INFO - Done with install on 172.23.120.206. 2021-03-02 00:14:37,052 - root - INFO - Done with install on 172.23.98.196. 2021-03-02 00:14:40,802 - root - INFO - Done with install on 172.23.121.10. 2021-03-02 00:14:41,490 - root - INFO - Done with install on 172.23.104.186. 2021-03-02 00:14:49,352 - root - INFO - Done with install on 172.23.98.195. 2021-03-02 00:15:01,150 - root - INFO - Setting KV memory quota as 1886 MB on 172.23.120.206 2021-03-02 00:15:01,387 - root - INFO - Done with init on 172.23.120.206. 2021-03-02 00:15:01,411 - root - INFO - Done with cleanup on 172.23.120.206. 2021-03-02 00:15:38,737 - root - INFO - Setting KV memory quota as 1887 MB on 172.23.98.196 2021-03-02 00:15:39,015 - root - INFO - Done with init on 172.23.98.196. 2021-03-02 00:15:39,065 - root - INFO - Done with cleanup on 172.23.98.196. 2021-03-02 00:15:42,572 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.121.10 2021-03-02 00:15:42,865 - root - INFO - Done with init on 172.23.121.10. 2021-03-02 00:15:42,920 - root - INFO - Done with cleanup on 172.23.121.10. 2021-03-02 00:15:43,141 - root - INFO - Setting KV memory quota as 3106 MB on 172.23.104.186 2021-03-02 00:15:43,387 - root - INFO - Done with init on 172.23.104.186. 2021-03-02 00:15:43,707 - root - INFO - Done with cleanup on 172.23.104.186. 2021-03-02 00:15:50,993 - root - INFO - Setting KV memory quota as 1887 MB on 172.23.98.195 2021-03-02 00:15:51,233 - root - INFO - Done with init on 172.23.98.195. 2021-03-02 00:15:51,269 - root - INFO - Done with cleanup on 172.23.98.195. 2021-03-02 00:15:57,295 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-02 00:15:57,334 - root - INFO - node:172.23.98.196:8091 version:7.0.0-4574-enterprise aFamily:inet services:[u'kv'] 2021-03-02 00:15:57,369 - root - INFO - node:172.23.98.195:8091 version:7.0.0-4574-enterprise aFamily:inet services:[u'kv'] 2021-03-02 00:15:57,408 - root - INFO - node:172.23.121.10:8091 version:7.0.0-4574-enterprise aFamily:inet services:[u'kv'] 2021-03-02 00:15:57,448 - root - INFO - node:172.23.104.186:8091 version:7.0.0-4574-enterprise aFamily:inet services:[u'kv'] 2021-03-02 00:15:57,487 - root - INFO - node:172.23.120.206:8091 version:7.0.0-4574-enterprise aFamily:inet services:[u'kv'] 2021-03-02 00:15:57,490 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-02 00:15:57,492 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-02 00:15:57,493 - root - INFO - INSTALL COMPLETED ON: 172.23.98.196 2021-03-02 00:15:57,493 - root - INFO - INSTALL COMPLETED ON: 172.23.98.195 2021-03-02 00:15:57,496 - root - INFO - INSTALL COMPLETED ON: 172.23.121.10 2021-03-02 00:15:57,496 - root - INFO - INSTALL COMPLETED ON: 172.23.104.186 2021-03-02 00:15:57,497 - root - INFO - INSTALL COMPLETED ON: 172.23.120.206 2021-03-02 00:15:57,499 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-02 00:15:57,507 - root - INFO - TOTAL INSTALL TIME = 472.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 4s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'cluster_name': 'win10-bucket-ops', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 5, 'quota_percent': '95', 'rebalance_moves_per_node': '64', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-Mar-02_00-16-12/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,rebalance_moves_per_node=64 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-Mar-02_00-16-12/test_1', 'nodes_init': '3', 'rebalance_moves_per_node': '64', 'GROUP': 'rebalance_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 1, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-02 00:16:23,759 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-02 00:16:25,144 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-02 00:16:27,575 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-02 00:17:05,332 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-02 00:17:23,734 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-03-02 00:17:23,773 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-02 00:17:23,822 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-02 00:17:36,871 | test | INFO | pool-1-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-4574-enterprise | 10.3232272613 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-03-02 00:17:36,904 | test | INFO | pool-1-thread-6 | [task:check:330] Rebalance - status: running, progress: 0 2021-03-02 00:17:47,026 | test | INFO | pool-1-thread-6 | [task:check:330] Rebalance - status: none, progress: 100 2021-03-02 00:17:47,062 | test | INFO | pool-1-thread-6 | [task:check:389] Rebalance completed with progress: 100% in 10.1909999847 sec 2021-03-02 00:17:47,486 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1790] Creating required buckets from template 2021-03-02 00:18:46,328 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2252] Waiting for all collections to be created 2021-03-02 00:18:46,799 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 629145600 | 473293648 | 255686025 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 629145600 | 270058040 | 102 | | default | couchbase | 3 | none | 0 | 0 | 6291456000 | 534664232 | 167713510 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-03-02 00:18:46,802 | test | INFO | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-03-02 00:19:22,950 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4645] Performing scope/collection specific operations 2021-03-02 00:19:22,997 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4735] Done Performing scope/collection specific operations 2021-03-02 00:19:57,934 | test | INFO | pool-1-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............*................................*............................................*.........X | 102.582 | | 81.310 | ..............................................................................*X | 81.310 | | 80.095 | .............................................................................*X | 80.095 | +--------+------------------------------------------------------------------------------------------------------------+---------+ 2021-03-02 00:19:59,680 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..........*...............*..........................*.......................................**.....*..*.*..X | 16524.555 | +-------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-02 00:20:00,368 | test | INFO | pool-1-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......................................**......................................*..............X | 93.785 | | 79.810 | .............................................................................**.............**.........X | 101.981 | +--------+----------------------------------------------------------------------------------------------------------+---------+ 2021-03-02 00:20:00,865 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 96.08190091 | 3693387776 | 2209366016 | 35467264 / 3909087232 | 125178 / 247013 | 7.0.0-4574-enterprise | | 172.23.98.195 | kv | 95.2477249747 | 3693387776 | 2220167168 | 42897408 / 3909087232 | 154720 / 303318 | 7.0.0-4574-enterprise | | 172.23.121.10 | kv | 99.95 | 4201836544 | 2643701760 | 2359296 / 3758092288 | 134204 / 77293 | 7.0.0-4574-enterprise | +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-02 00:20:05,841 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:13,355 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:20,688 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:25,262 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:28,048 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:34,265 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:35,384 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:43,278 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:20:52,061 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:21:01,082 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-02 00:21:06,086 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4402] Validating collection stats and item counts 2021-03-02 00:22:36,851 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 143988160 | 329478219 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 629145600 | 313926472 | 102 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 524102632 | 596750032 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-02 00:22:36,852 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-02 00:22:44,256 | test | INFO | MainThread | [collections_rebalance:setUp:56] Changing scrape interval to 5 2021-03-02 00:22:49,375 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 64} 2021-03-02 00:22:49,377 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:772] Doing collection data load before rebalance_in 2021-03-02 00:22:49,387 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4645] Performing scope/collection specific operations 2021-03-02 00:23:17,904 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4735] Done Performing scope/collection specific operations 2021-03-02 00:23:18,226 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:295] Starting rebalance operation of type : rebalance_in 2021-03-02 00:23:26,615 | test | ERROR | pool-1-thread-24 | [rest_client:_http_request:747] POST http://172.23.98.196:8091/controller/addNode body: hostname=http%3A%2F%2F172.23.104.186%3A8091&password=password&user=Administrator headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] auth: Administrator:password 2021-03-02 00:23:26,691 | test | INFO | pool-1-thread-24 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.98.196: 2021-03-02 00:23:26,694 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 5, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673406575L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:26.575Z', u'text': u'Failed to add node 172.23.104.186:8091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: "[\\"Unexpected server error, request logged.\\"]"'} 2021-03-02 00:23:26,696 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_cluster', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673406279L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:26.279Z', u'text': u'Add transaction of \'ns_1@172.23.104.186\' failed because of {error,\n complete_join,\n <<"Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {error,\n rest_error,\n <<"Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {bad_status,500,\n <<"[\\"Unexpected server error, request logged.\\"]">>}}}'} 2021-03-02 00:23:26,697 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.121.10', u'tstamp': 1614673400964L, u'shortText': u'node up', u'serverTime': u'2021-03-02T00:23:20.964Z', u'text': u"Node 'ns_1@172.23.121.10' saw that node 'ns_1@172.23.104.186' came up. Tags: []"} 2021-03-02 00:23:26,703 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.98.195', u'tstamp': 1614673400668L, u'shortText': u'node up', u'serverTime': u'2021-03-02T00:23:20.668Z', u'text': u"Node 'ns_1@172.23.98.195' saw that node 'ns_1@172.23.104.186' came up. Tags: []"} 2021-03-02 00:23:26,704 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_config', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673400517L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:20.517Z', u'text': u'Conflicting configuration changes to field scramsha_fallback_salt:\n<<94,224,205,108,43,158,127,81,243,212,191,200>> and\n<<189,227,193,55,25,180,200,157,148,81,60,14>>, choosing the former.'} 2021-03-02 00:23:26,706 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_config', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673400516L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:20.516Z', u'text': u'Conflicting configuration changes to field fts_memory_quota:\n256 and\n512, choosing the former.'} 2021-03-02 00:23:26,707 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_config', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673400515L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:20.515Z', u'text': u'Conflicting configuration changes to field cbas_memory_quota:\n1114 and\n1365, choosing the former.'} 2021-03-02 00:23:26,707 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673400503L, u'shortText': u'node up', u'serverTime': u'2021-03-02T00:23:20.503Z', u'text': u"Node 'ns_1@172.23.98.196' saw that node 'ns_1@172.23.104.186' came up. Tags: []"} 2021-03-02 00:23:26,709 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614673400116L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:23:20.116Z', u'text': u"Started node add transaction by adding node 'ns_1@172.23.104.186' to nodes_wanted (group: undefined)"} 2021-03-02 00:23:26,710 | test | ERROR | pool-1-thread-24 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.10', u'tstamp': 1614673355909L, u'shortText': u'message', u'serverTime': u'2021-03-02T00:22:35.909Z', u'text': u"Control connection to memcached on 'ns_1@172.23.121.10' disconnected. Check logs for details. (repeated 2 times, last seen 37.969317 secs ago)"} Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2611 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2611 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2613 2021-03-02 00:23:26,858 | test | ERROR | pool-1-thread-24 | [rest_client:add_node:1152] Add_node error: ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] FAIL 2021-03-02 00:23:26,858 | test | ERROR | pool-1-thread-24 | [task:call:242] Error adding node: 172.23.104.186 to the cluster:172.23.98.196 - ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.104.186:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] 2021-03-02 00:23:26,864 | test | INFO | MainThread | [collections_rebalance:tearDown:76] Reverting prometheus settings back to default 2021-03-02 00:23:31,368 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-03-02 00:23:33,115 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-02 00:23:33,117 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-02 00:24:35,354 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-02 00:24:35,355 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-02 00:25:29,586 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-02 00:25:29,588 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-02 00:25:34,348 | test | INFO | pool-1-thread-15 | [table_view:display:72] Keys failed in bucket2:_default:NdQ8jhLPWjTjkoRmpkqpBZ1U-QcMgdtEmLa-4s94Uxy-h7Rmb1eu9eYOMTNCphavnmwZgJIyGCWgCQKf461cSRdfwmo8PuU_BJAu9kyFyw6T-1BSigTM%qUxy5o1EJCNMnrPU-KNoRipNr82aeVGaeynANnpUI84FIKjJsB82sAAe9a9i7JBcAekS-47-549000 +---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | {'error': 'com.couchbase.client.core.error.UnambiguousTimeoutException | GetRequest, Reason: TIMEOUT | 172.23.98.195:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-16 | {'error': 'com.couchbase.client.core.error.UnambiguousTimeoutException | GetRequest, Reason: TIMEOUT | 172.23.98.196:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-03-02 00:25:34,351 | test | INFO | pool-1-thread-15 | [table_view:display:72] Keys failed in bucket2:_default:NdQ8jhLPWjTjkoRmpkqpBZ1U-QcMgdtEmLa-4s94Uxy-h7Rmb1eu9eYOMTNCphavnmwZgJIyGCWgCQKf461cSRdfwmo8PuU_BJAu9kyFyw6T-1BSigTM%qUxy5o1EJCNMnrPU-KNoRipNr82aeVGaeynANnpUI84FIKjJsB82sAAe9a9i7JBcAekS-47-549000 +---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | com.couchbase.client.core.error.AmbiguousTimeoutException | InsertRequest, Reason: TIMEOUT | 172.23.98.195:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED] | | test_collections-16 | com.couchbase.client.core.error.AmbiguousTimeoutException | InsertRequest, Reason: TIMEOUT | 172.23.98.196:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED] | +---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-03-02 00:25:34,355 | test | ERROR | pool-1-thread-27 | [task:call:3783] Failed to load 2 docs from 15 to 17 2021-03-02 00:25:35,200 | test | INFO | pool-1-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5164.602 | | 926.244 | ................*X | 926.244 | | 653.159 | ...........X | 653.159 | | 82.678 | ***X | 86.390 | | 51.795 | **X | 51.795 | | 17.595 | X | 17.595 | | 2.000 | *X | 2.000 | | 1.900 | ******X | 2.000 | | 2.000 | ***********X | 2.900 | | 2.500 | **X | 2.500 | | 1.900 | ********X | 2.000 | | 2.000 | *****X | 2.000 | +---------+---------------------------------------------------------------------------------------------------------+----------+ 2021-03-02 00:25:36,542 | test | INFO | pool-1-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **................................................................*...........................*..X | 13758.014 | | 13637.381 | ...........................................................................................*..*......X | 14732.097 | | 14249.365 | ...............................................................................................*X | 14249.365 | | 6449.177 | ..........................................X | 6449.177 | | 6449.177 | ..........................................X | 6449.177 | | 2313.377 | ..............X | 2313.377 | | 705.959 | ...X | 705.959 | | 0.000 | **************************************X | 0.000 | +-----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-03-02 00:25:37,470 | test | INFO | pool-1-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...........................................................................*........................X | 3204.201 | | 2968.274 | ...........................................................................................X | 2968.274 | | 2532.988 | ..............................................................................X | 2532.988 | | 1223.869 | .....................................*X | 1223.869 | | 610.366 | ..................*X | 610.366 | | 221.243 | .....X | 221.243 | | 41.497 | X | 41.497 | | 10.997 | *X | 10.997 | | 0.600 | X | 0.600 | | 0.000 | **************************************X | 0.000 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-03-02 00:26:30,525 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:718] 172.23.121.10: 1 core dump seen 2021-03-02 00:26:38,700 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:719] 172.23.121.10: Stack Trace of first crash - ae8d6778-2a62-426c-7fdd22bd-95275336.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f51691a3387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); #0 0x00007f51691a3387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 9893 selftid = 10119 #1 0x00007f51691a4a78 in __GI_abort () at abort.c:90 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0, 13575246249237626898, 0, 4613425733568182272, 0, 1, 139987636146755, 139987636130688, 139987632687742, 139987636146624, 10, 139986405952032, 8392704, 0, 139987632689011, 139987636146624}}, sa_flags = 1775231360, sa_restorer = 0x7f5169da43a2} sigs = {__val = {32, 0 }} #2 0x00007f5169d01195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x000000000055c972 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:93 No locals. #4 0x00007f5169cfef86 in __cxxabiv1::__terminate (handler=) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47 No locals. #5 0x00007f5169cfefd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57 No locals. #6 0x00007f5169d29dfe in std::execute_native_thread_routine (__p=0x7f5167c6e4d0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:91 __t = {_M_t = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x7f5167c6e4d0}, }, }}} #7 0x00007f5169542ea5 in start_thread (arg=0x7f51277fe700) at pthread_create.c:307 __res = pd = 0x7f51277fe700 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139986531772160, -7648902475751057947, 0, 8392704, 0, 139986531772160, 7745044868954625509, 7744908302283871717}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = #8 0x00007f516926b8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done. done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.17.so.debug...done. done. Loaded symbols for /lib64/librt.so.1 0x00007fe145218eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) Thread 32 (Thread 0x7fe1445ff700 (LWP 11442)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fe146fc35a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fe1445e6a60) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fe1445e6a60) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fe1454efea5 in start_thread (arg=0x7fe1445ff700) at pthread_create.c:307 #7 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7fe14387e700 (LWP 11443)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fe1487ed8bd in __gthread_cond_timedwait ( __abs_timeout=0x7fe143864fa0, __mutex=, __cond=0x7fe144a12538) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fe144a12538) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7fe144a12538) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7fe144a12538) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7fe144a12538) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7fe144a12510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7fe144a12510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #8 0x00007fe1487ede10 in spdlog::details::thread_pool::worker_loop_ ( this=0x7fe144a12510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #9 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe144a45200) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #10 0x00007fe1454efea5 in start_thread (arg=0x7fe14387e700) at pthread_create.c:307 #11 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7fe14307d700 (LWP 11444)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fe1487ecf7a in __gthread_cond_timedwait ( __abs_timeout=0x7fe1430641f0, __mutex=, __cond=) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7fe143c55988) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fe143c55988) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7fe143c55988) at /usr/local/include/c++/7.3.0/thread:243 #11 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0x7fe143c55980) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe143c55980) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #13 0x00007fe1454efea5 in start_thread (arg=0x7fe14307d700) at pthread_create.c:307 #14 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7fe14287c700 (LWP 11445)): #0 0x00007fe14520dc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000046af9e in get_command (buffsize=80, buffer=0x7fe1428631d0 "") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:49 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:90 #3 0x00007fe147cb7727 in run (this=0x7fe144b463f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #4 platform_thread_wrap (arg=0x7fe144b463f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #5 0x00007fe1454efea5 in start_thread (arg=0x7fe14287c700) at pthread_create.c:307 #6 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7fe14207b700 (LWP 11446)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe14908bb0b in consume_socket (thread_index=, sp=0xb4d490, ctx=0xb4bf70) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18718 #2 worker_thread_run (conn=0xb4cba0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18859 #3 worker_thread (thread_func_param=0xb4cba0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19002 #4 0x00007fe1454efea5 in start_thread (arg=0x7fe14207b700) at pthread_create.c:307 #5 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7fe14187a700 (LWP 11447)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe14908bb0b in consume_socket (thread_index=, sp=0xb4de78, ctx=0xb4bf70) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18718 #2 worker_thread_run (conn=0xb4d588) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18859 #3 worker_thread (thread_func_param=0xb4d588) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19002 #4 0x00007fe1454efea5 in start_thread (arg=0x7fe14187a700) at pthread_create.c:307 #5 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 0x7fe141079700 (LWP 11448)): #0 0x00007fe14520dc3d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fe149082ab8 in master_thread_run (ctx=0xb4bf70) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19148 #2 master_thread (thread_func_param=0xb4bf70) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19243 #3 0x00007fe1454efea5 in start_thread (arg=0x7fe141079700) at pthread_create.c:307 #4 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 0x7fe140878700 (LWP 11449)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000497270 in __gthread_cond_timedwait ( __abs_timeout=0x7fe14085f1d0, __mutex=, __cond=0x7fe143c640f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=, this=0x7fe143c640f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=, this=0x7fe143c640f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=, __lock=, this=0x7fe143c640f8) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 ExternalAuthManagerThread::run (this=0x7fe143c64000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:229 #6 0x00007fe147ccdc8f in Couchbase::Thread::thread_entry ( this=0x7fe143c64000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #7 0x00007fe147cb7727 in run (this=0x7fe144b46b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fe144b46b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fe1454efea5 in start_thread (arg=0x7fe140878700) at pthread_create.c:307 #10 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 0x7fe13bfff700 (LWP 11450)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000004f63a6 in __gthread_cond_timedwait ( __abs_timeout=0x7fe13bfe61f0, __mutex=, __cond=0x7fe143c698f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fe143c698f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > ( __atime=..., __lock=..., this=0x7fe143c698f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fe143c698f8) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 AuditImpl::consume_events (this=0x7fe143c69500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:408 #6 0x00007fe147cb7727 in run (this=0x7fe144b468a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #7 platform_thread_wrap (arg=0x7fe144b468a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #8 0x00007fe1454efea5 in start_thread (arg=0x7fe13bfff700) at pthread_create.c:307 #9 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7fe13b7fe700 (LWP 11451)): #0 cb::_JEArenaMalloc::free (ptr=0x7fe13025dc20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_malloc.cc:271 #1 0x00000000005283eb in free (ptr=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/cb_arena_malloc.h:170 #2 cb_free (ptr=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_malloc_arena.cc:73 #3 0x00007fe149e5b345 in operator delete (ptr=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/global_new_replacement.cc:98 #4 0x00007fe149055108 in deallocate (this=, __p=) at /usr/local/include/c++/7.3.0/ext/new_allocator.h:125 #5 deallocate (__a=..., __n=, __p=) at /usr/local/include/c++/7.3.0/bits/alloc_traits.h:462 #6 _M_destroy (__size=, this=0x7fe13b7e2f70) at /usr/local/include/c++/7.3.0/bits/basic_string.h:226 #7 _M_dispose (this=0x7fe13b7e2f70) at /usr/local/include/c++/7.3.0/bits/basic_string.h:221 #8 ~basic_string (this=0x7fe13b7e2f70, __in_chrg=) at /usr/local/include/c++/7.3.0/bits/basic_string.h:647 #9 CBStatCollector::addStat (this=0x7fe13b7e4860, k=..., v=..., labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/cbstat_collector.cc:40 #10 0x00007fe149054b0f in CBStatCollector::addStat (this=0x7fe13b7e4860, k=..., v=0, labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/cbstat_collector.cc:64 #11 0x00007fe1490671f1 in forwardToParent (labels=..., v=, k=..., this=, this=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/labelled_collector.h:134 #12 LabelledStatCollector::addStat (this=this@entry=0x7fe13b7e35c0, k=..., v=v@entry=0, labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/labelled_collector.cc:68 #13 0x00007fe1494daa19 in addStat (v=, labels=..., k=, this=0x7fe13b7e35c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/collector.h:323 #14 StatCollector::addStat(cb::stats::Key&&, unsigned long&&) const (this=this@entry=0x7fe13b7e35c0, v=, k=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/collector.h:333 #15 0x00007fe1494e845d in KVBucket::appendAggregatedVBucketStats ( this=this@entry=0x7fe13029e000, active=..., replica=..., pending=..., dead=..., collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1255 #16 0x00007fe1494a61bd in EphemeralBucket::appendAggregatedVBucketStats ( this=0x7fe13029e000, active=..., replica=..., pending=..., dead=..., collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ephemeral_bucket.cc:389 #17 0x00007fe1494dd301 in KVBucket::getAggregatedVBucketStats ( this=0x7fe13029e000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1198 #18 0x00007fe14947b2ae in EventuallyPersistentEngine::doEngineStats ( this=this@entry=0x7fe130200000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2920 #19 0x00007fe14948f3eb in EventuallyPersistentEngine::getStats(void const*, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=this@entry=0x7fe130200000, cookie=0x7fe0d9bf9700, key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4635 #20 0x00007fe149490267 in EventuallyPersistentEngine::get_stats(gsl::not_null, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=0x7fe130200000, cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:371 #21 0x000000000044f032 in bucket_get_stats(Cookie&, std::basic_string_view >, cb::sized_buffer, std::function >, std::basic_string_view >, gsl::not_null)> const&) ( cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:381 #22 0x00000000004d90da in stat_all_stats (arg=..., cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:490 #23 0x00000000004d8d5d in StatsCommandContext::doStats ( this=this@entry=0x7fe13809e6f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:727 #24 0x00000000004d9ea8 in StatsCommandContext::step (this=0x7fe13809e6f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:649 #25 0x00000000004dcc10 in SteppableCommandContext::drive ( this=this@entry=0x7fe13809e6f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:42 #26 0x00000000004b1b86 in stat_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:192 #27 0x00000000004b4ad6 in operator() (__args#0=..., this=) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #28 execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:920 #29 0x00000000004a2909 in Cookie::execute (this=0x7fe0d9bf9700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:158 #30 0x0000000000483b0d in Connection::executeCommandPipeline ( this=this@entry=0x7fe0d9bab700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:564 #31 0x000000000048ad75 in Connection::executeCommandsCallback ( this=this@entry=0x7fe0d9bab700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:690 #32 0x000000000048bc3f in Connection::rw_callback (ctx=0x7fe0d9bab700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:819 #33 0x00007fe146d77d8e in bufferevent_run_deferred_callbacks_unlocked ( cb=, arg=0x7fe0d9bf9980) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208 #34 0x00007fe146d80d01 in event_process_active_single_queue ( base=base@entry=0x7fe143c83300, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #35 0x00007fe146d8154f in event_process_active (base=0x7fe143c83300) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #36 event_base_loop (base=0x7fe143c83300, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #37 0x000000000046bd29 in worker_libevent (arg=0x7fe143cfb000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #38 0x00007fe147cb7727 in run (this=0x7fe144a19a40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #39 platform_thread_wrap (arg=0x7fe144a19a40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #40 0x00007fe1454efea5 in start_thread (arg=0x7fe13b7fe700) at pthread_create.c:307 #41 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7fe13affd700 (LWP 11452)): #0 0x00007fe145218eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fe146d8a3c5 in epoll_dispatch (base=0x7fe143c83600, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe146d81376 in event_base_loop (base=0x7fe143c83600, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000046bd29 in worker_libevent (arg=0x7fe143cfbaa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #4 0x00007fe147cb7727 in run (this=0x7fe144a19c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #5 platform_thread_wrap (arg=0x7fe144a19c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #6 0x00007fe1454efea5 in start_thread (arg=0x7fe13affd700) at pthread_create.c:307 #7 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7fe13a7fc700 (LWP 11453)): #0 0x00007fe146fb7f53 in tsd_fetch_impl (minimal=false, init=false) at include/jemalloc/internal/tsd.h:353 #1 tsdn_fetch () at include/jemalloc/internal/tsd.h:394 #2 je_nallocx (size=size@entry=25, flags=0) at src/jemalloc.c:3630 #3 0x00007fe147cd1178 in cb::JEArenaCoreLocalTracker::memAllocated ( index=1 '\001', size=size@entry=25, alignment=alignment@entry=0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_corelocal_tracker.cc:119 #4 0x00007fe147cd15f8 in memAllocated (alignment=0, size=25, index=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/je_arena_malloc.h:105 #5 cb::_JEArenaMalloc::malloc ( size=size@entry=25) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_malloc.cc:231 #6 0x00000000005282fe in malloc (size=25) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/cb_arena_malloc.h:140 #7 cb_malloc (size=size@entry=25) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_malloc_arena.cc:46 #8 0x00007fe149e5b289 in operator new (count=count@entry=25) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/global_new_replacement.cc:70 #9 0x00007fe149053393 in allocate (this=, __n=25) at /usr/local/include/c++/7.3.0/ext/new_allocator.h:111 #10 allocate (__a=..., __n=25) at /usr/local/include/c++/7.3.0/bits/alloc_traits.h:436 #11 _M_create (this=0x7fe13a7e1640, __old_capacity=0, __capacity=) at /usr/local/include/c++/7.3.0/bits/basic_string.tcc:153 #12 std::__cxx11::basic_string, std::allocator >::_M_construct (this=this@entry=0x7fe13a7e1640, __beg=0x7fe13a7e13d0 "vb_active_hp_vb_req_size-", __end=) at /usr/local/include/c++/7.3.0/bits/basic_string.tcc:219 #13 0x00007fe149054dba in _M_construct_aux ( __end=, __beg=, this=0x7fe13a7e1640) at /usr/local/include/c++/7.3.0/bits/basic_string.h:236 #14 _M_construct (__end=, __beg=, this=0x7fe13a7e1640) at /usr/local/include/c++/7.3.0/bits/basic_string.h:255 #15 basic_string (__a=..., __n=, __s=, this=0x7fe13a7e1640) at /usr/local/include/c++/7.3.0/bits/basic_string.h:502 #16 CBStatCollector::formatKey[abi:cxx11](std::basic_string_view >, std::unordered_map >, std::basic_string_view >, std::hash > >, std::equal_to > >, std::allocator > const, std::basic_string_view > > > > const&) const ( this=this@entry=0x7fe13a7e2860, key=..., labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/cbstat_collector.cc:257 #17 0x00007fe1490550a4 in CBStatCollector::addStat (this=0x7fe13a7e2860, k=..., v=..., labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/cbstat_collector.cc:40 #18 0x00007fe149054b0f in CBStatCollector::addStat (this=0x7fe13a7e2860, k=..., v=0, labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/cbstat_collector.cc:64 #19 0x00007fe1490671f1 in forwardToParent (labels=..., v=, k=..., this=, this=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/labelled_collector.h:134 #20 LabelledStatCollector::addStat (this=this@entry=0x7fe13a7e1c90, k=..., v=v@entry=0, labels=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/labelled_collector.cc:68 #21 0x00007fe1494daa19 in addStat (v=, labels=..., k=, this=0x7fe13a7e1c90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/collector.h:323 #22 StatCollector::addStat(cb::stats::Key&&, unsigned long&&) const (this=this@entry=0x7fe13a7e1c90, v=, k=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/statistics/collector.h:333 #23 0x00007fe1494e81e1 in KVBucket::appendAggregatedVBucketStats ( this=, active=..., replica=..., pending=..., dead=..., collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1226 #24 0x00007fe1494dd301 in KVBucket::getAggregatedVBucketStats ( this=0x7fe10a607300, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1198 #25 0x00007fe14947b2ae in EventuallyPersistentEngine::doEngineStats ( this=this@entry=0x7fe10a5fd000, collector=...) Build was aborted Aborted by Balakumaran G Archiving artifacts at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2920 #26 0x00007fe14948f3eb in EventuallyPersistentEngine::getStats(void const*, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=this@entry=0x7fe10a5fd000, cookie=0x7fe1310cb680, key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4635 #27 0x00007fe149490267 in EventuallyPersistentEngine::get_stats(gsl::not_null, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=0x7fe10a5fd000, cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:371 #28 0x000000000044f032 in bucket_get_stats(Cookie&, std::basic_string_view >, cb::sized_buffer, std::function >, std::basic_string_view >, gsl::not_null)> const&) ( cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:381 #29 0x00000000004d90da in stat_all_stats (arg=..., cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:490 #30 0x00000000004d8d5d in StatsCommandContext::doStats ( this=this@entry=0x7fe13116c5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:727 #31 0x00000000004d9ea8 in StatsCommandContext::step (this=0x7fe13116c5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:649 #32 0x00000000004dcc10 in SteppableCommandContext::drive ( this=this@entry=0x7fe13116c5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:42 #33 0x00000000004b1b86 in stat_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:192 #34 0x00000000004b4ad6 in operator() (__args#0=..., this=) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #35 execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:920 #36 0x00000000004a2909 in Cookie::execute (this=0x7fe1310cb680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:158 #37 0x0000000000483b0d in Connection::executeCommandPipeline ( this=this@entry=0x7fe1383c6f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:564 #38 0x000000000048ad75 in Connection::executeCommandsCallback ( this=this@entry=0x7fe1383c6f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:690 #39 0x000000000048bc3f in Connection::rw_callback (ctx=0x7fe1383c6f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:819 #40 0x00007fe146d77d8e in bufferevent_run_deferred_callbacks_unlocked ( cb=, arg=0x7fe1310cb900) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208 #41 0x00007fe146d80d01 in event_process_active_single_queue ( base=base@entry=0x7fe143c83900, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #42 0x00007fe146d8154f in event_process_active (base=0x7fe143c83900) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #43 event_base_loop (base=0x7fe143c83900, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #44 0x000000000046bd29 in worker_libevent (arg=0x7fe143cfc550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #45 0x00007fe147cb7727 in run (this=0x7fe144a19c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #46 platform_thread_wrap (arg=0x7fe144a19c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #47 0x00007fe1454efea5 in start_thread (arg=0x7fe13a7fc700) at pthread_create.c:307 #48 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7fe139ffb700 (LWP 11454)): #0 0x00007fe145218eb3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fe146d8a3c5 in epoll_dispatch (base=0x7fe143c83c00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe146d81376 in event_base_loop (base=0x7fe143c83c00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000046bd29 in worker_libevent (arg=0x7fe143cfcff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #4 0x00007fe147cb7727 in run (this=0x7fe144a19c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #5 platform_thread_wrap (arg=0x7fe144a19c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #6 0x00007fe1454efea5 in start_thread (arg=0x7fe139ffb700) at pthread_create.c:307 #7 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7fe1397fa700 (LWP 11455)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe145cd11bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe143b20158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fe143b20158, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7bc1 in Executor::run (this=0x7fe143b20000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fe147ccdc8f in Couchbase::Thread::thread_entry ( this=0x7fe143b20000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fe147cb7727 in run (this=0x7fe144a19bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fe144a19bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fe1454efea5 in start_thread (arg=0x7fe1397fa700) at pthread_create.c:307 #8 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7fe138ff9700 (LWP 11456)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe145cd11bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe143b20318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fe143b20318, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7bc1 in Executor::run (this=0x7fe143b201c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fe147ccdc8f in Couchbase::Thread::thread_entry ( this=0x7fe143b201c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fe147cb7727 in run (this=0x7fe144a19d10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fe144a19d10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fe1454efea5 in start_thread (arg=0x7fe138ff9700) at pthread_create.c:307 #8 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7fe133fff700 (LWP 11457)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe145cd11bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe143b204d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fe143b204d8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7bc1 in Executor::run (this=0x7fe143b20380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fe147ccdc8f in Couchbase::Thread::thread_entry ( this=0x7fe143b20380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fe147cb7727 in run (this=0x7fe144a19b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fe144a19b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fe1454efea5 in start_thread (arg=0x7fe133fff700) at pthread_create.c:307 #8 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7fe1337fe700 (LWP 11458)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe145cd11bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe143b20698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fe143b20698, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7bc1 in Executor::run (this=0x7fe143b20540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fe147ccdc8f in Couchbase::Thread::thread_entry ( this=0x7fe143b20540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fe147cb7727 in run (this=0x7fe144a19d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fe144a19d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fe1454efea5 in start_thread (arg=0x7fe1337fe700) at pthread_create.c:307 #8 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7fe10bfff700 (LWP 11460)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fe146fc2fa2 in background_thread_sleep (tsdn=, interval=, info=) at src/background_thread.c:232 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fe10bfe6a60) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fe1454efea5 in start_thread (arg=0x7fe10bfff700) at pthread_create.c:307 #6 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7fe10affd700 (LWP 11462)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fe146fc2e51 in background_thread_sleep (tsdn=, interval=1000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fe10afe4a60) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fe1454efea5 in start_thread (arg=0x7fe10affd700) at pthread_create.c:307 #6 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7fe10a5fc700 (LWP 11463)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27db018) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27db018, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe14962098d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27db018, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143d6e400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fe1494b7ec8 in operator() (this=0x7fe1382d3f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fe1382d3f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe1494b7da3 in operator() (this=0x7fe144a41810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fe144a41800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c7f30) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fe1454efea5 in start_thread (arg=0x7fe10a5fc700) at pthread_create.c:307 #25 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7fe109dfb700 (LWP 11464)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27db090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27db090, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe14962098d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27db090, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143d6e400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fe1494b7ec8 in operator() (this=0x7fe1382d3f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fe1382d3f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe1494b7da3 in operator() (this=0x7fe144a41860) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fe144a41850) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c7fa0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fe1454efea5 in start_thread (arg=0x7fe109dfb700) at pthread_create.c:307 #25 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7fe1095fa700 (LWP 11465)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27db060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27db060, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe14962098d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27db060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143d6e400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fe1494b7ec8 in operator() (this=0x7fe1382d3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fe1382d3fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe1494b7da3 in operator() (this=0x7fe144a435c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fe144a435b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c8010) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fe1454efea5 in start_thread (arg=0x7fe1095fa700) at pthread_create.c:307 #25 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7fe108df9700 (LWP 11466)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27db030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27db030, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe14962098d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27db030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143d6e400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fe1494b7ec8 in operator() (this=0x7fe138319040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fe138319040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe1494b7da3 in operator() (this=0x7fe144a43610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fe144a43600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c8080) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fe1454efea5 in start_thread (arg=0x7fe108df9700) at pthread_create.c:307 #25 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7fe0f27da700 (LWP 11467)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27dc200) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27dc200, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe149620ac7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27dc200, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc0018c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc0018c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc001880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143dda000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x00007fe1494b7ec8 in operator() (this=0x7fe138319140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #19 operator() (__closure=0x7fe138319140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fe1494b7da3 in operator() (this=0x7fe144a437a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fe144a43790) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c80f0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #25 0x00007fe1454efea5 in start_thread (arg=0x7fe0f27da700) at pthread_create.c:307 #26 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fe0f1fd9700 (LWP 11468)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27dc128) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27dc128, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe149620ac7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27dc128, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe0fc0018c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x00007fe149624750 in try_wait_for > ( timeout=, this=0x7fe0fc0018c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe0fc001880, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x00007fe1496193e7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe143dda000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x00007fe149633f39 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x00007fe1494b7ec8 in operator() (this=0x7fe138319180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #19 operator() (__closure=0x7fe138319180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fe1494b7da3 in operator() (this=0x7fe144a437f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fe144a437e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fe145cd6dcf in std::execute_native_thread_routine ( __p=0x7fe1310c8160) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #25 0x00007fe1454efea5 in start_thread (arg=0x7fe0f1fd9700) at pthread_create.c:307 #26 0x00007fe1452188dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fe0f17d8700 (LWP 11469)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fe149613e6d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fe14961b07a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe0f27dbfc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe0f27dbfc0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fe149620ac7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe0f27dbfc0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x00007fe149620ff6 in try_wait_until > > (opt=..., deadline=..., Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion temp_durability is disabled. Triggering skipped Finished: ABORTED