Details
-
Bug
-
Resolution: Fixed
-
Major
-
master
-
None
Description
Jepsen nightly 20190804T232636.000-0700 crashed during cluster setup with the following stacktrace.
2019-08-04 23:34:04,952{GMT} WARN [main] jepsen.core: Test crashed!
|
clojure.lang.ExceptionInfo: clj-http: status 400 {:cached nil, :request-time 897, :repeatable? false, :protocol-version {:name "HTTP", :major 1, :minor 1}, :streaming? true, :http-client #object[org.apache.http.impl.client.InternalHttpClient 0x7433c686 "org.apache.http.impl.client.InternalHttpClient@7433c686"], :chunked? false, :reason-phrase "Bad Request", :headers {"X-Permitted-Cross-Domain-Policies" "none", "Server" "Couchbase Server", "Content-Type" "application/json", "X-Content-Type-Options" "nosniff", "Content-Length" "177", "X-Frame-Options" "DENY", "Pragma" "no-cache", "Expires" "Thu, 01 Jan 1970 00:00:00 GMT", "Date" "Mon, 05 Aug 2019 06:28:58 GMT", "X-XSS-Protection" "1; mode=block", "Cache-Control" "no-cache,no-store,must-revalidate"}, :orig-content-encoding nil, :status 400, :length 177, :body "[\"Prepare join failed. Got HTTP status 500 from REST call post to https://172.28.128.253:18091/engageCluster2. Body was: \\\"[\\\\\\\"Unexpected server error, request logged.\\\\\\\"]\\\"\"]", :trace-redirects []}
|
at slingshot.support$stack_trace.invoke(support.clj:201) ~[knossos-0.3.4.jar:na]
|
at clj_http.client$exceptions_response.invokeStatic(client.clj:244) ~[na:na]
|
at clj_http.client$exceptions_response.invoke(client.clj:236) ~[na:na]
|
at clj_http.client$wrap_exceptions$fn__1978.invoke(client.clj:254) ~[na:na]
|
at clj_http.client$wrap_accept$fn__2224.invoke(client.clj:737) ~[na:na]
|
at clj_http.client$wrap_accept_encoding$fn__2231.invoke(client.clj:759) ~[na:na]
|
at clj_http.client$wrap_content_type$fn__2218.invoke(client.clj:720) ~[na:na]
|
at clj_http.client$wrap_form_params$fn__2327.invoke(client.clj:961) ~[na:na]
|
at clj_http.client$wrap_nested_params$fn__2348.invoke(client.clj:995) ~[na:na]
|
at clj_http.client$wrap_flatten_nested_params$fn__2357.invoke(client.clj:1019) ~[na:na]
|
at clj_http.client$wrap_method$fn__2285.invoke(client.clj:895) ~[na:na]
|
at clj_http.cookies$wrap_cookies$fn__413.invoke(cookies.clj:131) ~[na:na]
|
at clj_http.links$wrap_links$fn__899.invoke(links.clj:63) ~[na:na]
|
at clj_http.client$wrap_unknown_host$fn__2365.invoke(client.clj:1048) ~[na:na]
|
at clj_http.client$request_STAR_.invokeStatic(client.clj:1176) ~[na:na]
|
at clj_http.client$request_STAR_.invoke(client.clj:1169) ~[na:na]
|
at clj_http.client$post.invokeStatic(client.clj:1194) ~[na:na]
|
at clj_http.client$post.doInvoke(client.clj:1190) ~[na:na]
|
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.10.1.jar:na]
|
at couchbase.util$rest_call.invokeStatic(util.clj:23) ~[na:na]
|
at couchbase.util$rest_call.invoke(util.clj:15) ~[na:na]
|
at couchbase.util$rest_call.invokeStatic(util.clj:17) ~[na:na]
|
at couchbase.util$rest_call.invoke(util.clj:15) ~[na:na]
|
at couchbase.util$add_nodes.invokeStatic(util.clj:83) ~[na:na]
|
at couchbase.util$add_nodes.invoke(util.clj:73) ~[na:na]
|
at couchbase.util$add_nodes.invokeStatic(util.clj:75) ~[na:na]
|
at couchbase.util$add_nodes.invoke(util.clj:73) ~[na:na]
|
at couchbase.util$setup_cluster.invokeStatic(util.clj:297) ~[na:na]
|
at couchbase.util$setup_cluster.invoke(util.clj:285) ~[na:na]
|
at couchbase.core$couchbase$reify__4639.setup_primary_BANG_(core.clj:27) ~[na:na]
|
at jepsen.db$fn__2974$G__2970__2978.invoke(db.clj:12) ~[jepsen-0.1.14.jar:na]
|
at jepsen.db$fn__2974$G__2969__2983.invoke(db.clj:12) ~[jepsen-0.1.14.jar:na]
|
at clojure.core$partial$fn__5839.invoke(core.clj:2625) ~[clojure-1.10.1.jar:na]
|
at jepsen.control$on_nodes$fn__2918.invoke(control.clj:391) ~[jepsen-0.1.14.jar:na]
|
at clojure.lang.AFn.applyToHelper(AFn.java:154) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.1.jar:na]
|
at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.1.jar:na]
|
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.1.jar:na]
|
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.RestFn.applyTo(RestFn.java:142) ~[clojure-1.10.1.jar:na]
|
at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.1.jar:na]
|
at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.10.1.jar:na]
|
at dom_top.core$real_pmap_helper$build_thread__214$fn__215.invoke(core.clj:146) ~[jepsen-0.1.14.jar:na]
|
at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.10.1.jar:na]
|
at clojure.core$apply.invokeStatic(core.clj:665) ~[clojure-1.10.1.jar:na]
|
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973) ~[clojure-1.10.1.jar:na]
|
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.RestFn.invoke(RestFn.java:425) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.10.1.jar:na]
|
at clojure.core$apply.invokeStatic(core.clj:669) ~[clojure-1.10.1.jar:na]
|
at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.10.1.jar:na]
|
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.10.1.jar:na]
|
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222]
|
The babysitter log on 172.28.128.253 shows that node was repeatedly crashing during startup, triggering the above issue.
Attachments
Issue Links
- relates to
-
MB-35636 Jepsen crash in post-commit validation
- Closed