Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: master
Affects Version/s: master
Component/s: Jepsen
Labels:
None

Triage:
Untriaged
Is this a Regression?:
Unknown

Description

When running the kv-engine-jepsen-nightly-227 run jepsen crashed with the following stack trace when running test:
lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=set-kill --scenario=kill-memcached-on-slow-disk --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --manipulate-disks --enable-tcp-capture --hashdump --enable-memcached-debug-log-level &> jepsen-output-6.log

2019-08-22 00:18:22,301{GMT}	WARN	[jepsen worker 605] jepsen.core: Process 3605 crashed

java.util.concurrent.ExecutionException: com.couchbase.client.core.error.RequestTimeoutException: GetRequest {"retried":0,"reason":"TIMEOUT","requestId":7990,"timeoutMs":10000,"service":{"bucket":"default","scope":{"present":true},"collection":{"present":true},"type":"kv","key":"jepsen0000000004"},"cancelled":true,"coreId":3,"completed":true}

	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_222]

	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_222]

	at clojure.core$deref_future.invokeStatic(core.clj:2300) ~[clojure-1.10.1.jar:na]

	at clojure.core$future_call$reify__8454.deref(core.clj:6974) ~[clojure-1.10.1.jar:na]

	at clojure.core$deref.invokeStatic(core.clj:2320) ~[clojure-1.10.1.jar:na]

	at clojure.core$pmap$step__8467$fn__8471.invoke(core.clj:7025) ~[clojure-1.10.1.jar:na]

	at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.10.1.jar:na]

	at clojure.lang.LazySeq.seq(LazySeq.java:51) ~[clojure-1.10.1.jar:na]

	at clojure.lang.RT.seq(RT.java:535) ~[clojure-1.10.1.jar:na]

	at clojure.core$seq__5402.invokeStatic(core.clj:137) ~[clojure-1.10.1.jar:na]

	at clojure.core$filter$fn__5893.invoke(core.clj:2809) ~[clojure-1.10.1.jar:na]

	at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.10.1.jar:na]

	at clojure.lang.LazySeq.seq(LazySeq.java:51) ~[clojure-1.10.1.jar:na]

	at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.10.1.jar:na]

	at clojure.lang.RT.next(RT.java:713) ~[clojure-1.10.1.jar:na]

	at clojure.core$next__5386.invokeStatic(core.clj:64) ~[clojure-1.10.1.jar:na]

	at clojure.core$dorun.invokeStatic(core.clj:3142) ~[clojure-1.10.1.jar:na]

	at clojure.core$doall.invokeStatic(core.clj:3148) ~[clojure-1.10.1.jar:na]

	at clojure.core$doall.invoke(core.clj:3148) ~[clojure-1.10.1.jar:na]

	at couchbase.clients.NewSetClient.invoke_BANG_(clients.clj:350) ~[na:na]

	at jepsen.core$invoke_op_BANG_$fn__5516.invoke(core.clj:231) ~[jepsen-0.1.14.jar:na]

	at jepsen.core$invoke_op_BANG_.invokeStatic(core.clj:231) [jepsen-0.1.14.jar:na]

	at jepsen.core$invoke_op_BANG_.invoke(core.clj:226) [jepsen-0.1.14.jar:na]

	at jepsen.core.ClientWorker.run_worker_BANG_(core.clj:362) [jepsen-0.1.14.jar:na]

	at jepsen.core$run_workers_BANG_$run__5507.invoke(core.clj:214) [jepsen-0.1.14.jar:na]

	at dom_top.core$real_pmap_helper$build_thread__214$fn__215.invoke(core.clj:146) [jepsen-0.1.14.jar:na]

	at clojure.lang.AFn.applyToHelper(AFn.java:152) [clojure-1.10.1.jar:na]

	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.10.1.jar:na]

	at clojure.core$apply.invokeStatic(core.clj:665) [clojure-1.10.1.jar:na]

	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) [clojure-1.10.1.jar:na]

	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) [clojure-1.10.1.jar:na]

	at clojure.lang.RestFn.invoke(RestFn.java:425) [clojure-1.10.1.jar:na]

	at clojure.lang.AFn.applyToHelper(AFn.java:156) [clojure-1.10.1.jar:na]

	at clojure.lang.RestFn.applyTo(RestFn.java:132) [clojure-1.10.1.jar:na]

	at clojure.core$apply.invokeStatic(core.clj:669) [clojure-1.10.1.jar:na]

	at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003) [clojure-1.10.1.jar:na]

	at clojure.lang.RestFn.invoke(RestFn.java:397) [clojure-1.10.1.jar:na]

	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.10.1.jar:na]

	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]

Caused by: com.couchbase.client.core.error.RequestTimeoutException: GetRequest {"retried":0,"reason":"TIMEOUT","requestId":7990,"timeoutMs":10000,"service":{"bucket":"default","scope":{"present":true},"collection":{"present":true},"type":"kv","key":"jepsen0000000004"},"cancelled":true,"coreId":3,"completed":true}

	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:49) ~[java-client-3.0.0-alpha.6.jar:na]

	at com.couchbase.client.java.Collection.get(Collection.java:158) ~[java-client-3.0.0-alpha.6.jar:na]

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]

	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]

	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) ~[clojure-1.10.1.jar:na]

	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) ~[clojure-1.10.1.jar:na]

	at couchbase.clients$check_if_exists$fn__3254.invoke(clients.clj:310) ~[na:na]

	at couchbase.clients$check_if_exists.invokeStatic(clients.clj:310) ~[na:na]

	at couchbase.clients$check_if_exists.invoke(clients.clj:309) ~[na:na]

	at couchbase.clients.NewSetClient$fn__3279.invoke(clients.clj:348) ~[na:na]

	at clojure.core$pmap$fn__8462$fn__8463.invoke(core.clj:7022) ~[clojure-1.10.1.jar:na]

	at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030) ~[clojure-1.10.1.jar:na]

	at clojure.lang.AFn.call(AFn.java:18) [clojure-1.10.1.jar:na]

	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_222]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_222]

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_222]

	... 1 common frames omitted

	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.

		at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:131) ~[core-io-2.0.0-alpha.6.jar:na]

		at com.couchbase.client.core.Timer.lambda$register$1(Timer.java:101) ~[core-io-2.0.0-alpha.6.jar:na]

		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682) ~[core-io-2.0.0-alpha.6.jar:na]

		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757) ~[core-io-2.0.0-alpha.6.jar:na]

		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485) ~[core-io-2.0.0-alpha.6.jar:na]

		at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[core-io-2.0.0-alpha.6.jar:na]

		... 1 common frames omitted

2019-08-22 00:18:22,303{GMT}	INFO	[jepsen w

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

20190822T001626.000-0700.zip
14.42 MB
22/Aug/19 4:14 AM

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Sven Signer (Inactive)

Reporter:: Richard deMellow

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 22/Aug/19 4:14 AM

Updated:: 23/Aug/19 9:27 AM

Resolved:: 23/Aug/19 9:27 AM

Gerrit Reviews

There are no open Gerrit changes

Show There is 1 closed Gerrit change

Hide There is 1 closed Gerrit change

MB-35650 Improve timeout handling in set-workload: Gerrit Review:

Crash due to timeout when Jepsen is performing a GET

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty