Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8119

memcached.exe exits continuously in latest 2.0.2 builds

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: storage-engine
    • Security Level: Public
    • Labels:
      None

      Description

      SUBJ.

      2013-04-17 12:14:06.404 ns_memcached:1:info:message(ns_1@192.168.122.57) - Bucket "default" loaded on node 'ns_1@192.168.122.57' in 0 seconds. (repeated 5 times)
      2013-04-17 12:14:06.404 ns_memcached:4:info:message(ns_1@192.168.122.57) - Control connection to memcached on 'ns_1@192.168.122.57' disconnected: {badmatch,
      {error,
      closed}} (repeated 5 times)
      2013-04-17 12:14:06.544 ns_log:0:info:message(ns_1@192.168.122.57) - Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: Wed Apr 17 12:13:57.013800 Pacific Standard Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
      Wed Apr 17 12:13:57.014800 Pacific Standard Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
      Wed Apr 17 12:13:57.023800 Pacific Standard Time 3: (default) Failed to load mutation log, falling back to key dump
      Wed Apr 17 12:13:57.023800 Pacific Standard Time 3: Extension support isn't implemented in this version of bucket_engine
      Wed Apr 17 12:13:57.024800 Pacific Standard Time 3: (default) metadata loaded in 4897 usec
      Wed Apr 17 12:13:57.026800 Pacific Standard Time 3: (default) warmup completed in 5955 usec
      2013-04-17 12:14:06.544 ns_memcached:4:info:message(ns_1@192.168.122.57) - Control connection to memcached on 'ns_1@192.168.122.57' disconnected: {badmatch,
      {error,
      closed}}
      2013-04-17 12:14:07.059 ns_memcached:1:info:message(ns_1@192.168.122.57) - Bucket "default" loaded on node 'ns_1@192.168.122.57' in 0 seconds.
      2013-04-17 12:14:16.544 ns_log:0:info:message(ns_1@192.168.122.57) - Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: Wed Apr 17 12:14:07.057800 Pacific Standard Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
      Wed Apr 17 12:14:07.057800 Pacific Standard Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
      Wed Apr 17 12:14:07.062800 Pacific Standard Time 3: (default) Failed to load mutation log, falling back to key dump
      Wed Apr 17 12:14:07.062800 Pacific Standard Time 3: Extension support isn't implemented in this version of bucket_engine
      Wed Apr 17 12:14:07.062800 Pacific Standard Time 3: (default) metadata loaded in 1700 usec
      Wed Apr 17 12:14:07.063800 Pacific Standard Time 3: (default) warmup completed in 2193 usec

      See full collectinfo (with memcached logs) attached.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        trond Trond Norbye added a comment -

        It looks like it is a bug in the script that creates the memcached logs, or is all of these files empty?

        trond@ok:1022> cat memcached.log ~/Downloads/a/cbcollect_info_ns_1@192.168.122.57_20130417-201328
        ==============================================================================
        Memcached logs
        cd "c:\Program Files\Couchbase\Server\var\lib\couchbase\logs" && for /f %a IN ('dir /od /b memcached.log.*') do type %a
        ==============================================================================

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.6.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.7.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.1.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.2.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.8.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.9.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.3.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.4.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.10.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.11.txt

        c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.0.txt

        — end —

        Show
        trond Trond Norbye added a comment - It looks like it is a bug in the script that creates the memcached logs, or is all of these files empty? trond@ok:1022> cat memcached.log ~/Downloads/a/cbcollect_info_ns_1@192.168.122.57_20130417-201328 ============================================================================== Memcached logs cd "c:\Program Files\Couchbase\Server\var\lib\couchbase\logs" && for /f %a IN ('dir /od /b memcached.log.*') do type %a ============================================================================== c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.6.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.7.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.1.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.2.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.8.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.9.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.3.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.4.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.10.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.11.txt c:\Program Files\Couchbase\Server\var\lib\couchbase\logs>type memcached.log.0.txt — end —
        Hide
        trond Trond Norbye added a comment -

        Mike: There hasn't been any other change in memcached lately except increase of a log level for an error message so I don't think this problem is caused by memcached. I think its more likely it is a change happening in ep_engine or libcouchstore etc..

        Show
        trond Trond Norbye added a comment - Mike: There hasn't been any other change in memcached lately except increase of a log level for an error message so I don't think this problem is caused by memcached. I think its more likely it is a change happening in ep_engine or libcouchstore etc..
        Hide
        mikew Mike Wiederhold added a comment -

        Aaron,

        Please take a look at this as soon as possible. It looks like a regression that is causing windows to crash continuously and this was seen right after you merged your compactor changes. Below is a stack trace.

        (gdb) bt
        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        #0 ?? (warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        ) at src/base/spinlock.h:97
        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        #1 0x0000000071102f32 in cfs_destroy (h=warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        0x5b071e0)
        at src/couch-kvstore/couch-fs-stats.cc:99
        #2 0x0000000071190ff3 in buffered_destructor ()
        from c:\Program Files\Couchbase\Server\lib\memcached\ep.so
        #3 0x000000007118db23 in tree_file_close ()
        from c:\Program Files\Couchbase\Server\lib\memcached\ep.so
        #4 0x000000007118a536 in couchstore_close_db ()
        from c:\Program Files\Couchbase\Server\lib\memcached\ep.so
        #5 0x000000007118a519 in couchstore_open_db_ex ()
        from c:\Program Files\Couchbase\Server\lib\memcached\ep.so
        #6 0x00000000710fb245 in CouchKVStore::openDB (this=0x5acc000,
        vbucketId=1023, fileRev=1, db=0x6a2f710, options=1, newFileRev=0x6a2f708)
        at src/couch-kvstore/couch-kvstore.cc:1129
        #7 0x00000000710fb75f in CouchKVStore::setVBucketState (this=0x5acc000,
        vbucketId=1023, vbstate=..., vb_change_type=1, newfile=false, notify=true)
        at src/couch-kvstore/couch-kvstore.cc:706
        #8 0x00000000710ff20d in CouchKVStore::snapshotVBuckets (this=0x5acc000,
        vbstates=...) at src/couch-kvstore/couch-kvstore.cc:596
        #9 0x00000000711167c5 in EventuallyPersistentStore::snapshotVBuckets (
        this=0x5aca000, priority=...) at src/ep.cc:845
        #10 0x00000000711c1151 in SnapshotVBucketsCallback::callback (
        this=<value optimized out>) at src/ep.cc:198
        #11 0x000000007110fad6 in Dispatcher::run (this=0x5af6000)
        at src/dispatcher.cc:171
        #12 0x00000000711108c9 in launch_dispatcher_thread (arg=0x5af6000)
        at src/dispatcher.cc:28
        #13 0x0000000062484e4e in pthread_timechange_handler_np ()
        from c:\Program Files\Couchbase\Server\bin\pthreadGC2.dll
        #14 0x0000000005af6000 in ?? ()
        #15 0x000000006ea3394c in __DTOR_LIST_ ()
        from c:\Program Files\Couchbase\Server\bin\libtcmalloc_minimal-4.dll
        #16 0x0000000000000000 in ?? ()
        warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

        (gdb) Quit (expect signal SIGINT when the program is resumed)
        (gdb) quit
        A debugging session is active.

        Inferior 1 [process 3708] will be detached.

        Quit anyway? (y or n) y
        Detaching from program: c:\Program Files\Couchbase\Server\bin\memcached.exe, Pid
        3708

        c:\mingw64\bin>�

        Show
        mikew Mike Wiederhold added a comment - Aaron, Please take a look at this as soon as possible. It looks like a regression that is causing windows to crash continuously and this was seen right after you merged your compactor changes. Below is a stack trace. (gdb) bt warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) #0 ?? (warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) ) at src/base/spinlock.h:97 warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) #1 0x0000000071102f32 in cfs_destroy (h=warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) 0x5b071e0) at src/couch-kvstore/couch-fs-stats.cc:99 #2 0x0000000071190ff3 in buffered_destructor () from c:\Program Files\Couchbase\Server\lib\memcached\ep.so #3 0x000000007118db23 in tree_file_close () from c:\Program Files\Couchbase\Server\lib\memcached\ep.so #4 0x000000007118a536 in couchstore_close_db () from c:\Program Files\Couchbase\Server\lib\memcached\ep.so #5 0x000000007118a519 in couchstore_open_db_ex () from c:\Program Files\Couchbase\Server\lib\memcached\ep.so #6 0x00000000710fb245 in CouchKVStore::openDB (this=0x5acc000, vbucketId=1023, fileRev=1, db=0x6a2f710, options=1, newFileRev=0x6a2f708) at src/couch-kvstore/couch-kvstore.cc:1129 #7 0x00000000710fb75f in CouchKVStore::setVBucketState (this=0x5acc000, vbucketId=1023, vbstate=..., vb_change_type=1, newfile=false, notify=true) at src/couch-kvstore/couch-kvstore.cc:706 #8 0x00000000710ff20d in CouchKVStore::snapshotVBuckets (this=0x5acc000, vbstates=...) at src/couch-kvstore/couch-kvstore.cc:596 #9 0x00000000711167c5 in EventuallyPersistentStore::snapshotVBuckets ( this=0x5aca000, priority=...) at src/ep.cc:845 #10 0x00000000711c1151 in SnapshotVBucketsCallback::callback ( this=<value optimized out>) at src/ep.cc:198 #11 0x000000007110fad6 in Dispatcher::run (this=0x5af6000) at src/dispatcher.cc:171 #12 0x00000000711108c9 in launch_dispatcher_thread (arg=0x5af6000) at src/dispatcher.cc:28 #13 0x0000000062484e4e in pthread_timechange_handler_np () from c:\Program Files\Couchbase\Server\bin\pthreadGC2.dll #14 0x0000000005af6000 in ?? () #15 0x000000006ea3394c in __ DTOR_LIST _ () from c:\Program Files\Couchbase\Server\bin\libtcmalloc_minimal-4.dll #16 0x0000000000000000 in ?? () warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.) (gdb) Quit (expect signal SIGINT when the program is resumed) (gdb) quit A debugging session is active. Inferior 1 [process 3708] will be detached. Quit anyway? (y or n) y Detaching from program: c:\Program Files\Couchbase\Server\bin\memcached.exe, Pid 3708 c:\mingw64\bin>�
        Hide
        maria Maria McDuff (Inactive) added a comment -

        tony, pls verify / close.

        Show
        maria Maria McDuff (Inactive) added a comment - tony, pls verify / close.

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            alkondratenko Aleksey Kondratenko (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes