Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35650

Crash due to timeout when Jepsen is performing a GET

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • master
    • master
    • Jepsen
    • None
    • Untriaged
    • Unknown

    Description

      When running the kv-engine-jepsen-nightly-227 run jepsen crashed with the following stack trace when running test:
      lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=set-kill --scenario=kill-memcached-on-slow-disk --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --manipulate-disks --enable-tcp-capture --hashdump --enable-memcached-debug-log-level &> jepsen-output-6.log

      2019-08-22 00:18:22,301{GMT}	WARN	[jepsen worker 605] jepsen.core: Process 3605 crashed
      java.util.concurrent.ExecutionException: com.couchbase.client.core.error.RequestTimeoutException: GetRequest {"retried":0,"reason":"TIMEOUT","requestId":7990,"timeoutMs":10000,"service":{"bucket":"default","scope":{"present":true},"collection":{"present":true},"type":"kv","key":"jepsen0000000004"},"cancelled":true,"coreId":3,"completed":true}
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_222]
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_222]
      	at clojure.core$deref_future.invokeStatic(core.clj:2300) ~[clojure-1.10.1.jar:na]
      	at clojure.core$future_call$reify__8454.deref(core.clj:6974) ~[clojure-1.10.1.jar:na]
      	at clojure.core$deref.invokeStatic(core.clj:2320) ~[clojure-1.10.1.jar:na]
      	at clojure.core$pmap$step__8467$fn__8471.invoke(core.clj:7025) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.LazySeq.seq(LazySeq.java:51) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.RT.seq(RT.java:535) ~[clojure-1.10.1.jar:na]
      	at clojure.core$seq__5402.invokeStatic(core.clj:137) ~[clojure-1.10.1.jar:na]
      	at clojure.core$filter$fn__5893.invoke(core.clj:2809) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.LazySeq.seq(LazySeq.java:51) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.RT.next(RT.java:713) ~[clojure-1.10.1.jar:na]
      	at clojure.core$next__5386.invokeStatic(core.clj:64) ~[clojure-1.10.1.jar:na]
      	at clojure.core$dorun.invokeStatic(core.clj:3142) ~[clojure-1.10.1.jar:na]
      	at clojure.core$doall.invokeStatic(core.clj:3148) ~[clojure-1.10.1.jar:na]
      	at clojure.core$doall.invoke(core.clj:3148) ~[clojure-1.10.1.jar:na]
      	at couchbase.clients.NewSetClient.invoke_BANG_(clients.clj:350) ~[na:na]
      	at jepsen.core$invoke_op_BANG_$fn__5516.invoke(core.clj:231) ~[jepsen-0.1.14.jar:na]
      	at jepsen.core$invoke_op_BANG_.invokeStatic(core.clj:231) [jepsen-0.1.14.jar:na]
      	at jepsen.core$invoke_op_BANG_.invoke(core.clj:226) [jepsen-0.1.14.jar:na]
      	at jepsen.core.ClientWorker.run_worker_BANG_(core.clj:362) [jepsen-0.1.14.jar:na]
      	at jepsen.core$run_workers_BANG_$run__5507.invoke(core.clj:214) [jepsen-0.1.14.jar:na]
      	at dom_top.core$real_pmap_helper$build_thread__214$fn__215.invoke(core.clj:146) [jepsen-0.1.14.jar:na]
      	at clojure.lang.AFn.applyToHelper(AFn.java:152) [clojure-1.10.1.jar:na]
      	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.10.1.jar:na]
      	at clojure.core$apply.invokeStatic(core.clj:665) [clojure-1.10.1.jar:na]
      	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) [clojure-1.10.1.jar:na]
      	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) [clojure-1.10.1.jar:na]
      	at clojure.lang.RestFn.invoke(RestFn.java:425) [clojure-1.10.1.jar:na]
      	at clojure.lang.AFn.applyToHelper(AFn.java:156) [clojure-1.10.1.jar:na]
      	at clojure.lang.RestFn.applyTo(RestFn.java:132) [clojure-1.10.1.jar:na]
      	at clojure.core$apply.invokeStatic(core.clj:669) [clojure-1.10.1.jar:na]
      	at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003) [clojure-1.10.1.jar:na]
      	at clojure.lang.RestFn.invoke(RestFn.java:397) [clojure-1.10.1.jar:na]
      	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.10.1.jar:na]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]
      Caused by: com.couchbase.client.core.error.RequestTimeoutException: GetRequest {"retried":0,"reason":"TIMEOUT","requestId":7990,"timeoutMs":10000,"service":{"bucket":"default","scope":{"present":true},"collection":{"present":true},"type":"kv","key":"jepsen0000000004"},"cancelled":true,"coreId":3,"completed":true}
      	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:49) ~[java-client-3.0.0-alpha.6.jar:na]
      	at com.couchbase.client.java.Collection.get(Collection.java:158) ~[java-client-3.0.0-alpha.6.jar:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) ~[clojure-1.10.1.jar:na]
      	at couchbase.clients$check_if_exists$fn__3254.invoke(clients.clj:310) ~[na:na]
      	at couchbase.clients$check_if_exists.invokeStatic(clients.clj:310) ~[na:na]
      	at couchbase.clients$check_if_exists.invoke(clients.clj:309) ~[na:na]
      	at couchbase.clients.NewSetClient$fn__3279.invoke(clients.clj:348) ~[na:na]
      	at clojure.core$pmap$fn__8462$fn__8463.invoke(core.clj:7022) ~[clojure-1.10.1.jar:na]
      	at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030) ~[clojure-1.10.1.jar:na]
      	at clojure.lang.AFn.call(AFn.java:18) [clojure-1.10.1.jar:na]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_222]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_222]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_222]
      	... 1 common frames omitted
      	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
      		at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:131) ~[core-io-2.0.0-alpha.6.jar:na]
      		at com.couchbase.client.core.Timer.lambda$register$1(Timer.java:101) ~[core-io-2.0.0-alpha.6.jar:na]
      		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682) ~[core-io-2.0.0-alpha.6.jar:na]
      		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757) ~[core-io-2.0.0-alpha.6.jar:na]
      		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485) ~[core-io-2.0.0-alpha.6.jar:na]
      		at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[core-io-2.0.0-alpha.6.jar:na]
      		... 1 common frames omitted
      2019-08-22 00:18:22,303{GMT}	INFO	[jepsen w
      

      Attachments

        For Gerrit Dashboard: MB-35650
        # Subject Branch Project Status CR V

        Activity

          Assigned to Sven Signer as I believe its related to the last change you made.

          richard.demellow Richard deMellow added a comment - Assigned to Sven Signer as I believe its related to the last change you made.

          People

            sven.signer Sven Signer (Inactive)
            richard.demellow Richard deMellow
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty