Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3495

memcached crashed after vbucketmigrator killed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 1.6.5.2
    • 1.6.5.2
    • couchbase-bucket
    • Security Level: Public
    • None
    • Ubuntu 1004 64bit

    Description

      Install ms 1.6.5.2r-13 on node 52.
      Load 6M keys with 2K per key to node 52
      Install ms 1.6.5.2r-13 on node 53, 55 and 56.
      Add node 53 to node 52
      Rebalance.
      After rebalance's done, kill vbuckmigrator several times. Replication worked as exptected
      After replication is done, add node 55 and 56 to cluster of node 52 and 53.
      Rebalance
      After rebalance's done, kill vbucketmigrator serveral times
      Then watch -d './stats 10.2.1.52:11211 all' shows error.
      http://screencast.com/t/cDxOHgozz
      Check /tmp dir, got core dump

      Thread 1 (Thread 13638):
      #0 reserve_cookie (cookie=0x0) at memcached.c:6121
      #1 0x00007f62df56d32c in TapConnection (this=0x10b505c00,
      theEngine=<value optimized out>, c=<value optimized out>,
      n=<value optimized out>, f=<value optimized out>) at tapconnection.cc:68
      #2 0x00007f62df5742b9 in TapConnMap::newConn (this=0x246b5a0,
      cookie=0x51ef8b48, name=..., flags=20, backfillAge=0, tapKeepAlive=300)
      at tapconnmap.cc:162
      #3 0x00007f62df54a9db in EventuallyPersistentEngine::createTapQueue (
      this=0x246b000, cookie=0x51ef8b48, client=<value optimized out>, flags=20,
      userdata=0x7f62e004e7fa, nuserdata=140062641940472) at ep_engine.cc:1634
      #4 0x00007f62df54ac58 in EvpGetTapIterator (handle=0x246b000,
      cookie=0x51ef8b48, client=<value optimized out>,
      nclient=<value optimized out>, flags=20, userdata=0x815ec04d,
      nuserdata=126) at ep_engine.cc:681
      #5 0x00007f62e20584bc in bucket_get_tap_iterator (handle=0x7f62e225d160,
      cookie=0x51ef8b48, client=0x815ec01c, nclient=49, flags=20,
      userdata=0x815ec04d, nuserdata=126) at bucket_engine.c:1122
      #6 0x0000000000408883 in process_bin_tap_connect (c=0x51ef8b48)
      at memcached.c:2592
      #7 0x0000000000408a19 in conn_setup_tap_stream (c=0x0) at memcached.c:5419
      #8 0x000000000041394e in libevent_tap_process (fd=<value optimized out>,
      which=<value optimized out>, arg=0x2376900) at thread.c:485
      #9 0x000000000041ca89 in event_process_active_single_queue (base=0x23a8800,
      flags=<value optimized out>) at event.c:1288
      #10 event_process_active (base=0x23a8800, flags=<value optimized out>)
      at event.c:1355
      #11 event_base_loop (base=0x23a8800, flags=<value optimized out>)
      at event.c:1550
      #12 0x0000000000414364 in worker_libevent (arg=0x2376900) at thread.c:254
      #13 0x00007f62e3f989ca in start_thread () from /lib/libpthread.so.0
      #14 0x00007f62e3cf669d in clone () from /lib/libc.so.6
      #15 0x0000000000000000 in ?? ()
      --------------------------------------------------------------------------------
      Module information:
      /opt/membase/lib/libtcmalloc_minimal.so.0:
      lrwxrwxrwx 1 membase membase 28 2011-03-07 16:17 /opt/membase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.0.0
      f860d74660d78069e3c8719ed91f98a3 /opt/membase/lib/libtcmalloc_minimal.so.0
      /lib/librt.so.1:
      lrwxrwxrwx 1 root root 15 2010-12-29 17:24 /lib/librt.so.1 -> librt-2.11.1.so
      1cfe1e6a4c1630a2e050303652c3341d /lib/librt.so.1
      /lib/libdl.so.2:
      lrwxrwxrwx 1 root root 15 2010-12-29 17:24 /lib/libdl.so.2 -> libdl-2.11.1.so
      4fc2038540e79999571aa4a338ac8692 /lib/libdl.so.2
      /lib/libpthread.so.0:
      lrwxrwxrwx 1 root root 20 2010-12-29 17:24 /lib/libpthread.so.0 -> libpthread-2.11.1.so
      deb7c9592f790b5ec2910b627df2d480 /lib/libpthread.so.0
      /lib/libc.so.6:
      lrwxrwxrwx 1 root root 14 2010-12-29 17:24 /lib/libc.so.6 -> libc-2.11.1.so
      df0cc88c32da17856164f20b4b82c54a /lib/libc.so.6
      /lib/libm.so.6:
      lrwxrwxrwx 1 root root 14 2010-12-29 17:24 /lib/libm.so.6 -> libm-2.11.1.so
      54f6f084ce7ce998d5ae8efaf5daea70 /lib/libm.so.6
      /usr/lib/libstdc++.so.6:
      lrwxrwxrwx 1 root root 19 2010-12-29 17:24 /usr/lib/libstdc+.so.6 -> libstdc+.so.6.0.13
      145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6
      /lib/libgcc_s.so.1:
      rw-rr- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1
      2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1
      /lib64/ld-linux-x86-64.so.2:
      lrwxrwxrwx 1 root root 12 2010-12-29 17:24 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so
      b319f05500878fd836406046864771fa /lib64/ld-linux-x86-64.so.2
      ./bin/memcached/stdin_term_handler.so:
      rw-rr- 1 membase membase 17142 2011-03-07 14:52 ./bin/memcached/stdin_term_handler.so
      7d30ad20749f9ebe46c0a6e388036f8d ./bin/memcached/stdin_term_handler.so
      ./bin/bucket_engine/bucket_engine.so:
      rw-rr- 1 membase membase 196697 2011-03-07 14:52 ./bin/bucket_engine/bucket_engine.so
      854552d20fbead0efb15c9d8238a39e3 ./bin/bucket_engine/bucket_engine.so
      bin/ep_engine/ep.so:
      rw-rr- 1 membase membase 11180416 2011-03-07 14:52 bin/ep_engine/ep.so
      83bedc8255d883d45e039bddb0ba8b83 bin/ep_engine/ep.so
      (END)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            trond Trond Norbye
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty