Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5403

Memcached crash post offline upgrade from 180 to latest 181

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 1.8.1
    • 1.8.1-release-candidate
    • couchbase-bucket
    • Security Level: Public
    • 1.8.1-846-rel

    Description

      Test case failing:-

      upgradetests.MultipleNodeUpgradeTests.test_upgrade,start_upgraded_first=True,insert_data=True,initial_version=1.7.0,create_buckets=True,upgrade_path=1.7.1.1,1.7.2,1.8.0r-55,online_upgrade=False,load_ratio=4 -p version=1.8.1-846-rel

      warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff0c7fd000
      Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
      Program terminated with signal 6, Aborted.
      #0 0x00000037f7630265 in raise () from /lib64/libc.so.6
      (gdb) t a a bt

      Thread 9 (Thread 0x2aff79b57220 (LWP 18421)):
      #0 0x00000037f76d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002aff796d9be8 in epoll_dispatch (base=0xd976000, tv=<value optimized out>) at epoll.c:404
      #2 0x00002aff796c7d5a in event_base_loop (base=0xd976000, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000408cd0 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7495

      Thread 8 (Thread 18422):
      #0 0x00000037f76c678b in read () from /lib64/libc.so.6
      #1 0x00000037f766cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x00000037f766d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x00000037f7668fdb in getc () from /lib64/libc.so.6
      #4 0x00002aff79b58875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19
      #5 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #6 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 7 (Thread 18423):
      #0 0x00000037f769a541 in nanosleep () from /lib64/libc.so.6
      #1 0x00000037f769a364 in sleep () from /lib64/libc.so.6
      #2 0x0000000000415cb8 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
      #3 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #4 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 6 (Thread 18424):
      #0 0x00000037f820d524 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x00000037f8208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
      #2 0x00000037f8208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00002aaaaaaae64c in must_lock (name=0x421c9a40 "bucket-1") at bucket_engine.c:252
      #4 lock_engines (name=0x421c9a40 "bucket-1") at bucket_engine.c:282
      #5 find_bucket (name=0x421c9a40 "bucket-1") at bucket_engine.c:568
      #6 0x00002aaaaaaae774 in handle_select_bucket (handle=0x2aaaaacb37e0, cookie=0xd993608, request=0xd997000, response=0x409460 <binary_response_handler>) at bucket_engine.c:2236
      #7 0x00002aaaaaaaf2c7 in bucket_unknown_command (handle=0x2aaaaacb37e0, cookie=0xd993608, request=0xd997000, response=0x409460 <binary_response_handler>) at bucket_engine.c:2325
      #8 0x000000000040ea5e in process_bin_unknown_packet (c=0xd993608) at daemon/memcached.c:2597
      #9 process_bin_packet (c=0xd993608) at daemon/memcached.c:2836
      #10 0x00000000004107dc in complete_nread (c=0x2aaaaacb3918) at daemon/memcached.c:3393
      #11 conn_nread (c=0x2aaaaacb3918) at daemon/memcached.c:5290
      #12 0x00000000004063fe in event_handler (fd=51, which=<value optimized out>, arg=0xd993608) at daemon/memcached.c:5532
      #13 0x00002aff796c8109 in event_base_loop (base=0xd976500, flags=<value optimized out>) at event.c:1308
      #14 0x0000000000413734 in worker_libevent (arg=0xd94e500) at daemon/thread.c:301
      #15 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #16 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 5 (Thread 18425):
      #0 0x00000037f76d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002aff796d9be8 in epoll_dispatch (base=0xd976280, tv=<value optimized out>) at epoll.c:404
      #2 0x00002aff796c7d5a in event_base_loop (base=0xd976280, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000413734 in worker_libevent (arg=0xd94e5f8) at daemon/thread.c:301
      #4 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 4 (Thread 18426):
      #0 0x00000037f820d524 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x00000037f8208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
      #2 0x00000037f8208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00002aaaaaaae505 in must_lock (handle=0x2aaaaacb37e0, cookie=0xd992dc8, request=<value optimized out>, response=0x409460 <binary_response_handler>) at bucket_engine.c:252
      #4 lock_engines (handle=0x2aaaaacb37e0, cookie=0xd992dc8, request=<value optimized out>, response=0x409460 <binary_response_handler>) at bucket_engine.c:282
      #5 handle_create_bucket (handle=0x2aaaaacb37e0, cookie=0xd992dc8, request=<value optimized out>, response=0x409460 <binary_response_handler>) at bucket_engine.c:2046
      #6 0x00002aaaaaaaf273 in bucket_unknown_command (handle=0x2aaaaacb37e0, cookie=0xd992dc8, request=0xd9bb800, response=0x409460 <binary_response_handler>) at bucket_engine.c:2313
      #7 0x000000000040ea5e in process_bin_unknown_packet (c=0xd992dc8) at daemon/memcached.c:2597
      #8 process_bin_packet (c=0xd992dc8) at daemon/memcached.c:2836
      #9 0x00000000004107dc in complete_nread (c=0x2aaaaacb3918) at daemon/memcached.c:3393
      #10 conn_nread (c=0x2aaaaacb3918) at daemon/memcached.c:5290
      #11 0x00000000004063fe in event_handler (fd=49, which=<value optimized out>, arg=0xd992dc8) at daemon/memcached.c:5532
      --Type <return> to continue, or q <return> to quit--
      #12 0x00002aff796c8109 in event_base_loop (base=0xd976c80, flags=<value optimized out>) at event.c:1308
      #13 0x0000000000413734 in worker_libevent (arg=0xd94e6f0) at daemon/thread.c:301
      #14 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #15 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 3 (Thread 18428):
      #0 0x00000037f76d48a8 in epoll_wait () from /lib64/libc.so.6
      #1 0x00002aff796d9be8 in epoll_dispatch (base=0xd976780, tv=<value optimized out>) at epoll.c:404
      #2 0x00002aff796c7d5a in event_base_loop (base=0xd976780, flags=<value optimized out>) at event.c:1558
      #3 0x0000000000413734 in worker_libevent (arg=0xd94e8e0) at daemon/thread.c:301
      #4 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #5 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 2 (Thread 18502):
      #0 0x00000037f769a541 in nanosleep () from /lib64/libc.so.6
      #1 0x00000037f76cded4 in usleep () from /lib64/libc.so.6
      #2 0x00002aaaaad51818 in updateStatsThread (arg=<value optimized out>) at memory_tracker.cc:31
      #3 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #4 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x43fcd940 (LWP 18427)):

      #0 0x00000037f7630265 in raise () from /lib64/libc.so.6
      #1 0x00000037f7631d10 in abort () from /lib64/libc.so.6
      #2 0x000000380a0bed14 in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
      #3 0x000000380a0bce16 in ?? () from /usr/lib64/libstdc++.so.6
      #4 0x000000380a0bce43 in std::terminate() () from /usr/lib64/libstdc++.so.6
      #5 0x000000380a0bcf2a in __cxa_throw () from /usr/lib64/libstdc++.so.6
      #6 0x00002aaaaad81753 in SqliteStrategy::open (this=<value optimized out>) at sqlite-strategies.cc:184
      #7 0x00002aaaaad7a99f in open (this=0xebd5d40, st=..., s=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string
      return '%s (count %d) %s' % (self.typename,
      TypeError: int argument required
      ) at sqlite-kvstore.hh:189
      #8 StrategicSqlite3::StrategicSqlite3 (this=0xebd5d40, st=..., s=Traceback (most recent call last):
      File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string
      return '%s (count %d) %s' % (self.typename,
      TypeError: int argument required
      ) at sqlite-kvstore.cc:18
      #9 0x00002aaaaad785a0 in KVStore::create (type=<value optimized out>, stats=..., conf=...) at kvstore.cc:53
      #10 0x00002aaaaad26100 in EventuallyPersistentEngine::newKVStore (this=<value optimized out>) at ep_engine.cc:1710
      #11 0x00002aaaaad0eb05 in EventuallyPersistentStore::EventuallyPersistentStore (this=0xebc2280, theEngine=..., t=<value optimized out>, startVb0=false, concurrentDB=true) at ep.cc:344
      #12 0x00002aaaaad3710e in EventuallyPersistentEngine::initialize (this=0xd9ee000,
      config=0x43fcc8c3 "ht_size=3079;ht_locks=5;db_shards=4;tap_noop_interval=20;max_txn_size=1000;max_size=1076887552;initfile=/opt/couchbase/etc/couchbase/init.sql;tap_keepalive=300;dbname=/opt/membase/var/lib/membase/data"...)
      at ep_engine.cc:1613
      #13 0x00002aaaaad388c6 in EvpInitialize (handle=0xd9ee000,
      config_str=0x43fcc8c3 "ht_size=3079;ht_locks=5;db_shards=4;tap_noop_interval=20;max_txn_size=1000;max_size=1076887552;initfile=/opt/couchbase/etc/couchbase/init.sql;tap_keepalive=300;dbname=/opt/membase/var/lib/membase/data"...)
      at ep_engine.cc:124
      #14 0x00002aaaaaaade33 in create_bucket_UNLOCKED (e=0x2aaaaacb37e0, bucket_name=0x43fcca30 "bucket-0", path=0x43fcc8a0 "/opt/couchbase/lib/memcached/ep.so",
      config=0x43fcc8c3 "ht_size=3079;ht_locks=5;db_shards=4;tap_noop_interval=20;max_txn_size=1000;max_size=1076887552;initfile=/opt/couchbase/etc/couchbase/init.sql;tap_keepalive=300;dbname=/opt/membase/var/lib/membase/data"...,
      e_out=0x0, msg=0x43fcc490 "", msglen=1024) at bucket_engine.c:685
      #15 0x00002aaaaaaae536 in handle_create_bucket (handle=0x2aaaaacb37e0, cookie=0xd993088, request=<value optimized out>, response=0x409460 <binary_response_handler>) at bucket_engine.c:2048
      #16 0x00002aaaaaaaf273 in bucket_unknown_command (handle=0x2aaaaacb37e0, cookie=0xd993088, request=0xd9bd000, response=0x409460 <binary_response_handler>) at bucket_engine.c:2313
      #17 0x000000000040ea5e in process_bin_unknown_packet (c=0xd993088) at daemon/memcached.c:2597
      #18 process_bin_packet (c=0xd993088) at daemon/memcached.c:2836
      #19 0x00000000004107dc in complete_nread (c=0x47f5) at daemon/memcached.c:3393
      #20 conn_nread (c=0x47f5) at daemon/memcached.c:5290
      #21 0x00000000004063fe in event_handler (fd=50, which=<value optimized out>, arg=0xd993088) at daemon/memcached.c:5532
      #22 0x00002aff796c8109 in event_base_loop (base=0xd976a00, flags=<value optimized out>) at event.c:1308
      #23 0x0000000000413734 in worker_libevent (arg=0xd94e7e8) at daemon/thread.c:301
      #24 0x00000037f820673d in start_thread () from /lib64/libpthread.so.0
      #25 0x00000037f76d44bd in clone () from /lib64/libc.so.6

      Attachments

        1. 10.1.3.114-8091-diag.txt.gz
          472 kB
        2. 10.1.3.114-8091-diag.txt.gz
          632 kB
        3. 10.1.3.115-8091-diag.txt.gz
          400 kB
        4. 10.1.3.115-8091-diag.txt.gz
          585 kB
        5. 10.1.3.116-8091-diag.txt.gz
          419 kB
        6. 10.1.3.116-8091-diag.txt.gz
          528 kB
        7. 10.1.3.117-8091-diag.txt.gz
          358 kB
        8. 10.1.3.117-8091-diag.txt.gz
          1.41 MB
        9. 10.1.3.118-8091-diag.txt.gz
          547 kB
        10. 10.1.3.118-8091-diag.txt.gz
          501 kB
        11. logs_117.zip
          3.47 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            karan Karan Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty