Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7264

Erlang crash on one node in a bidirectional XDCR set up (uptime 5 days)

    Details

      Description

      Cluster setup: c1:c2::10:10
      biXDCR_bucket: c1 <---> c2
      uniXDCR_src: c1 ---> c2 :uniXDCR_dest
      Front end loads on c1 and c2 for biXDCR_bucket, and on c1 for uniXDCR_src.
      c1: http://ec2-177-71-230-72.sa-east-1.compute.amazonaws.com:8091/
      c2: http://ec2-175-41-186-167.ap-southeast-1.compute.amazonaws.com:8091/

      Erlang crash on one node (c2): ec2-54-251-24-122.ap-southeast-1.compute.amazonaws.com
      Erlang core and erl

      Access: ssh -i SingaporeQAKey.pem ubuntu@ec2-54-251-24-122.ap-southeast-1.compute.amazonaws.com

      ubuntu@ip-10-135-47-251:/opt/couchbase/var/lib/couchbase$ sudo file core
      core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -S 16:16 -sbt u -P 327680 -K'

      ubuntu@ip-10-135-47-251:/opt/couchbase/var/lib/couchbase$ sudo gdb /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp core
      GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
      Copyright (C) 2012 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://bugs.launchpad.net/gdb-linaro/>...
      Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
      BFD: Warning: /opt/couchbase/var/lib/couchbase/core is truncated: expected core file size >= 1096171520, found: 123731968.
      [New LWP 15203]
      [New LWP 15207]
      [New LWP 15210]
      [New LWP 15208]
      [New LWP 15205]
      [New LWP 15216]
      [New LWP 15209]
      [New LWP 15215]
      [New LWP 15214]
      [New LWP 15213]
      [New LWP 15217]
      [New LWP 15206]
      [New LWP 15212]
      [New LWP 15211]
      [New LWP 15202]
      [New LWP 15201]
      [New LWP 15204]
      [New LWP 15200]
      [New LWP 15226]
      [New LWP 15197]
      [New LWP 15199]
      [New LWP 15198]
      Cannot access memory at address 0x7f52941962a8
      Cannot access memory at address 0x7f52941962a0

      (gdb) t a a bt

      Thread 22 (LWP 15198):
      #0 0x00007f5293441d2d in ?? ()
      Cannot access memory at address 0x7f52927f2e40
      (gdb) quit

      Core: https://s3.amazonaws.com/bugdb/MB--/core.tar
      Erl_crash_dump: https://s3.amazonaws.com/bugdb/MB--/erl_crash.dump.tar

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        abhinav Abhinav Dangeti added a comment -

        Note: couchbase buckets and swap space all set at data_path /mnt on the ec2 node (an additional mounted disk).
        The disk /mnt doesn't exist anymore (thus no bucket data available on the node) :: Not sure how this happened!

        Show
        abhinav Abhinav Dangeti added a comment - Note: couchbase buckets and swap space all set at data_path /mnt on the ec2 node (an additional mounted disk). The disk /mnt doesn't exist anymore (thus no bucket data available on the node) :: Not sure how this happened!
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - - edited

        Warning: /opt/couchbase/var/lib/couchbase/core is truncated: expected core file size >= 1096171520, found: 123731968.

        This is indication of not yet written core file. Next time please wait until it's complete

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - - edited Warning: /opt/couchbase/var/lib/couchbase/core is truncated: expected core file size >= 1096171520, found: 123731968. This is indication of not yet written core file. Next time please wait until it's complete
        Hide
        steve Steve Yen added a comment -

        from bug-scrub...

        assigning back to Abhinav to please try to reproduce

        Show
        steve Steve Yen added a comment - from bug-scrub... assigning back to Abhinav to please try to reproduce
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        Any update on reproducing this issue? Thanks.

        Show
        junyi Junyi Xie (Inactive) added a comment - Any update on reproducing this issue? Thanks.

          People

          • Assignee:
            abhinav Abhinav Dangeti
            Reporter:
            abhinav Abhinav Dangeti
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes