Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38305

Jepsen tests crash due to cluster setup receiving a http 500 error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 6.5.1, Cheshire-Cat
    • ns_server
    • Untriaged
    • No

    Description

      Jepsen tests fail as Jepsen framework is unable add nodes into a cluster. Crash occurs as the rest call to add nodes made from the first node (172.23.105.3) throws an exception and returns a HTTP status 500 error with Unexpected Server error.

      An example of a failed Jepsen test -
      http://qa.sc.couchbase.com/job/jepsen-durability-failover-daily-new/115/consoleFull

      lein trampoline run test --nodes-file ./nodes --username root --password couchbase --package ./couchbase-server-enterprise-7.0.0-1576-centos7.x86_64.rpm  --workload=failover --node-count=6 --no-autofailover --replicas=1 --failover-type=hard --recovery-type=full --disrupt-count=1 --kv-timeout=1.5 --durability=0:100:0:0 --doc-count=4000 --doc-threads=1 &> jepsen-output-1.log
      

      To reproduce this manually:
      1. Spin up 2 nodes with vagrant - 10.112.194.101 & 10.112.194.102 with the above server-build.

      2. Setup a new one node cluster with 10.112.194.101

      3. Add the second node to this cluster

       curl -u Administrator:abc123 \10.112.194.101:8091/controller/addNode \-d 'hostname=10.112.194.102&user=Administrator&password=abc123'
      

      Gives, "
      ["Prepare join failed. Got HTTP status 500 from REST call post to https://10.112.194.102:18091/engageCluster2. Body was: \"\\\"Unexpected server error, request logged.\\\"\""] "

       

      Sanity for this server build has failed. http://server.jenkins.couchbase.com/job/build_sanity_matrix/7191/DISTRO=suse12,TYPE=4node/consoleText

      Attaching the ns-server error log, and jepsen-logs and a few screenshots (all from Jepsen test)

      Attachments

        1. jepsen_output_2.png
          jepsen_output_2.png
          563 kB
        2. jepsen_output.png
          jepsen_output.png
          498 kB
        3. latest.zip
          12.84 MB
        4. ns-server_error.png
          ns-server_error.png
          434 kB
        5. Screen Shot 2020-03-16 at 3.38.44 PM.png
          Screen Shot 2020-03-16 at 3.38.44 PM.png
          90 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            timofey.barmin Timofey Barmin
            sumedh.basarkod Sumedh Basarkod (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty