Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.1
    • Component/s: moxi
    • Security Level: Public
    • Environment:
      client side moxi version 1.7.0
      cluster version 1.7.1
      moxi command line:
      3 nodes
      centos 5.4 64bit

      Description

      1) Set up 3 node cluster
      2) Start up moxi
      /opt/moxi/bin/moxi -v http://10.1.5.41:8091/pools/default/bucketsStreaming/default,http://10.1.5.42:8091/pools/default/bucketsStreaming/default,http://10.1.5.134:8091/pools/default/bucketsStreaming/default -Z port_listen=11211 -u root -t 16
      3) Start up set/get load against cluster
      4) Start up rebalance in a loop (continually add/remove node)
      5) load 1000 keys to be used for multiget
      6) run ascii multiget in a loop

      moxi: cproxy.c:927: cproxy_reserve_downstream: Assertion `d->timeout_tv.tv_sec == 0' failed.

      Thread 19 (process 24286):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f3baf0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f3baf0, flags=0) at event.c:1558
      #3 0x0000000000408da6 in main (argc=9, argv=0x7fff83c83bf8) at memcached.c:5061

      Thread 18 (process 24287):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f40640, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f40640, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 17 (process 24288):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f44c30, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f44c30, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 16 (process 24289):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f49220, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f49220, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 15 (process 24290):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f4d910, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f4d910, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 14 (process 24291):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f52000, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f52000, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 13 (process 24292):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f566f0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f566f0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 12 (process 24293):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5b150, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5b150, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 11 (process 24294):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5f8d0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5f8d0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 10 (process 24295):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f641c0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f641c0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 9 (process 24296):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f68ab0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f68ab0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 8 (process 24297):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f6d3a0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f6d3a0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 7 (process 24298):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f71c90, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f71c90, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 6 (process 24299):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f76580, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f76580, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 5 (process 24301):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f7fb60, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f7fb60, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 4 (process 24302):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f84850, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f84850, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 3 (process 24303):
      #0 0x0000003e6000ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x0000000000412a50 in assoc_maintenance_thread (arg=<value optimized out>) at assoc.c:220
      #2 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #3 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 2 (process 24304):
      #0 0x0000003e5f4cae46 in poll () from /lib64/libc.so.6
      #1 0x000000000044fc56 in Curl_socket_ready ()
      #2 0x000000000046671b in Curl_do_perform ()
      #3 0x000000000044993b in run_rest_conflate (arg=0x8f8b930) at rest.c:293
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 1 (process 24300):
      #0 0x0000003e5f430265 in raise () from /lib64/libc.so.6
      #1 0x0000003e5f431d10 in abort () from /lib64/libc.so.6
      #2 0x0000003e5f4296e6 in __assert_fail () from /lib64/libc.so.6
      #3 0x000000000041a5ad in cproxy_reserve_downstream (ptd=0x90bbe00) at cproxy.c:927
      #4 0x000000000041a7ae in cproxy_assign_downstream (ptd=0x90bbe00) at cproxy.c:1783
      #5 0x000000000040c86b in drive_machine (c=0x92f8600) at memcached.c:1988
      #6 0x0000000000442321 in event_base_loop (base=0x8f7ae70, flags=0) at event.c:1308
      #7 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #8 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      1. MB-4334_multi_ascii.py
        1 kB
        Steve Yen
      2. MB-4334_rebalance.sh
        0.2 kB
        Steve Yen
      3. MB-4334-moxi-vvv.txt
        720 kB
        Steve Yen

        Activity

        Hide
        steve Steve Yen added a comment -

        reopening to get it out of the current sprint

        Show
        steve Steve Yen added a comment - reopening to get it out of the current sprint
        Hide
        steve Steve Yen added a comment -

        Test using Keith's scripts has been running for several hours now with the fix, without a crash. It only stopped because somebody reset the cluster.

        Show
        steve Steve Yen added a comment - Test using Keith's scripts has been running for several hours now with the fix, without a crash. It only stopped because somebody reset the cluster.
        Hide
        steve Steve Yen added a comment -
        Show
        steve Steve Yen added a comment - http://review.couchbase.org/15232
        Hide
        steve Steve Yen added a comment -

        Added scripts from Keith B. and moxi -vvv output leading up to the crash

        Show
        steve Steve Yen added a comment - Added scripts from Keith B. and moxi -vvv output leading up to the crash
        Hide
        steve Steve Yen added a comment -

        Replicated crash on OSX moxi 1.8.0-3-g5f01fa8...

        Process: moxi [9936]
        Path: /Users/steveyen/work/membase/dev/voltron.couchbase/build/moxi/moxi
        Identifier: moxi
        Version: ??? (???)
        Code Type: X86-64 (Native)
        Parent Process: bash [8289]

        Date/Time: 2012-04-25 15:38:34.254 -0700
        OS Version: Mac OS X 10.6.8 (10K549)
        Report Version: 6

        Exception Type: EXC_CRASH (SIGABRT)
        Exception Codes: 0x0000000000000000, 0x0000000000000000
        Crashed Thread: 4

        Application Specific Information:
        Assertion failed: (d->timeout_tv.tv_sec == 0), function cproxy_reserve_downstream, file cproxy.c, line 927.

        Thread 0: Dispatch queue: com.apple.main-thread
        0 libSystem.B.dylib 0x00007fff87c41c0a kevent + 10
        1 moxi 0x000000010004be07 kq_dispatch + 423
        2 moxi 0x0000000100046080 event_base_loop + 576
        3 moxi 0x0000000100005ac3 main + 3891 (memcached.c:5063)
        4 moxi 0x0000000100001504 start + 52

        Thread 1:
        0 libSystem.B.dylib 0x00007fff87c3b272 write + 10
        1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230)
        2 moxi 0x000000010000a282 drive_machine + 242 (memcached.c:3456)
        3 moxi 0x00000001000462c2 event_base_loop + 1154
        4 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274)
        5 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        6 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 2:
        0 libSystem.B.dylib 0x00007fff87c3b272 write + 10
        1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230)
        2 moxi 0x0000000100001c40 conn_set_state + 160 (memcached.c:668)
        3 moxi 0x0000000100026c5d a2b_process_downstream_response + 2397 (cproxy_protocol_a2b.c:764)
        4 moxi 0x000000010000ac0c drive_machine + 2684 (memcached.c:1990)
        5 moxi 0x00000001000462c2 event_base_loop + 1154
        6 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274)
        7 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        8 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 3:
        0 libSystem.B.dylib 0x00007fff87c3b272 write + 10
        1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230)
        2 moxi 0x0000000100027a0e cproxy_process_a2b_downstream_nread + 654 (cproxy_protocol_a2b.c:541)
        3 moxi 0x000000010000ac0c drive_machine + 2684 (memcached.c:1990)
        4 moxi 0x00000001000462c2 event_base_loop + 1154
        5 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274)
        6 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        7 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 4 Crashed:
        0 libSystem.B.dylib 0x00007fff87c770b6 __kill + 10
        1 libSystem.B.dylib 0x00007fff87d179f6 abort + 83
        2 libSystem.B.dylib 0x00007fff87d049bc __pthread_markcancel + 0
        3 moxi 0x000000010001a5b7 cproxy_reserve_downstream + 839 (cproxy.c:926)
        4 moxi 0x000000010001a774 cproxy_assign_downstream + 196 (cproxy.c:1785)
        5 moxi 0x0000000100021905 cproxy_process_upstream_ascii + 901 (cproxy_protocol_a.c:150)
        6 moxi 0x0000000100008083 try_read_command + 835 (memcached.c:3205)
        7 moxi 0x000000010000a560 drive_machine + 976 (memcached.c:3536)
        8 moxi 0x00000001000462c2 event_base_loop + 1154
        9 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274)
        10 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        11 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 5:
        0 libSystem.B.dylib 0x00007fff87c63a6a __semwait_signal + 10
        1 libSystem.B.dylib 0x00007fff87c67881 _pthread_cond_wait + 1286
        2 moxi 0x0000000100011941 assoc_maintenance_thread + 305 (assoc.c:223)
        3 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        4 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 6:
        0 libSystem.B.dylib 0x00007fff87c6c932 select$DARWIN_EXTSN + 10
        1 libconflate.0.dylib 0x00000001000c2996 Curl_socket_ready + 438
        2 libconflate.0.dylib 0x00000001000b67f0 Curl_do_perform + 496
        3 libconflate.0.dylib 0x0000000100096719 run_rest_conflate + 801 (rest.c:293)
        4 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331
        5 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13

        Thread 4 crashed with X86 Thread State (64-bit):
        rax: 0x0000000000000000 rbx: 0x0000000100053f05 rcx: 0x0000000100509ac8 rdx: 0x0000000000000000
        rdi: 0x00000000000026d0 rsi: 0x0000000000000006 rbp: 0x0000000100509ae0 rsp: 0x0000000100509ac8
        r8: 0x0000000000000003 r9: 0x0000000000000000 r10: 0x00007fff87c730fa r11: 0x0000000000000206
        r12: 0x000000000000039f r13: 0x000000010005230f r14: 0x0000000100052e36 r15: 0x00007fff70c88790
        rip: 0x00007fff87c770b6 rfl: 0x0000000000000206 cr2: 0x0000000102997000

        Binary Images:
        0x100000000 - 0x100064fef +moxi ??? (???) <64AE645C-76DA-41FB-A00C-C4EB4DDFE5D9> /Users/steveyen/work/membase/dev/voltron.couchbase/build/moxi/moxi
        0x100093000 - 0x1000d9fef +libconflate.0.dylib ??? (???) <A2698F74-7010-74E0-F8BF-AB40EA051667> /opt/couchbase/lib/libconflate.0.dylib
        0x1000e7000 - 0x1000e9ff7 +libhashkit.0.dylib ??? (???) <EAF5A8E0-FD91-4E2E-07AC-E891EAE0F003> /opt/couchbase/lib/libhashkit.0.dylib
        0x1000ed000 - 0x1000f3fef +libvbucket.1.dylib 3.0.0 (compatibility 3.0.0) <2581DCBA-8D50-E90A-ECAF-372830B80C0D> /opt/couchbase/lib/libvbucket.1.dylib
        0x1000f9000 - 0x100109fef +libmemcached.6.dylib 7.0.0 (compatibility 7.0.0) <58E35902-BFC2-DF19-5802-7FAC3E201773> /opt/couchbase/lib/libmemcached.6.dylib
        0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???) <69130DA3-7CB3-54C8-ABC5-423DECDD2AF7> /usr/lib/dyld
        0x7fff813a2000 - 0x7fff813a6ff7 libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /usr/lib/system/libmathCommon.A.dylib
        0x7fff85634000 - 0x7fff85645ff7 libz.1.dylib 1.2.3 (compatibility 1.0.0) <97019C74-161A-3488-41EC-A6CA8738418C> /usr/lib/libz.1.dylib
        0x7fff87c28000 - 0x7fff87de9fef libSystem.B.dylib 125.2.11 (compatibility 1.0.0) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib
        0x7fff8835d000 - 0x7fff8839afff com.apple.LDAPFramework 2.0 (120.1) <54A6769E-D7E2-DBE2-EA61-87B9EA355DA4> /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
        0x7fff88499000 - 0x7fff884abfe7 libsasl2.2.dylib 3.15.0 (compatibility 3.0.0) <76B83C8D-8EFE-4467-0F75-275648AFED97> /usr/lib/libsasl2.2.dylib
        0x7fff884ef000 - 0x7fff88510fff libresolv.9.dylib 41.1.0 (compatibility 1.0.0) <9410EC7F-4D24-6740-AFEE-90405750FAD7> /usr/lib/libresolv.9.dylib
        0x7fff8a461000 - 0x7fff8a580fe7 libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <14115D29-432B-CF02-6B24-A60CC533A09E> /usr/lib/libcrypto.0.9.8.dylib
        0x7fff8a6b3000 - 0x7fff8a6b4ff7 com.apple.TrustEvaluationAgent 1.1 (1) <5952A9FA-BC2B-16EF-91A7-43902A5C07B6> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
        0x7fff8a6f7000 - 0x7fff8a734ff7 libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <F743389F-F25A-A77D-4FCA-D6B01AF2EE6D> /usr/lib/libssl.0.9.8.dylib
        0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib

        Show
        steve Steve Yen added a comment - Replicated crash on OSX moxi 1.8.0-3-g5f01fa8... Process: moxi [9936] Path: /Users/steveyen/work/membase/dev/voltron.couchbase/build/moxi/moxi Identifier: moxi Version: ??? (???) Code Type: X86-64 (Native) Parent Process: bash [8289] Date/Time: 2012-04-25 15:38:34.254 -0700 OS Version: Mac OS X 10.6.8 (10K549) Report Version: 6 Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Crashed Thread: 4 Application Specific Information: Assertion failed: (d->timeout_tv.tv_sec == 0), function cproxy_reserve_downstream, file cproxy.c, line 927. Thread 0: Dispatch queue: com.apple.main-thread 0 libSystem.B.dylib 0x00007fff87c41c0a kevent + 10 1 moxi 0x000000010004be07 kq_dispatch + 423 2 moxi 0x0000000100046080 event_base_loop + 576 3 moxi 0x0000000100005ac3 main + 3891 (memcached.c:5063) 4 moxi 0x0000000100001504 start + 52 Thread 1: 0 libSystem.B.dylib 0x00007fff87c3b272 write + 10 1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230) 2 moxi 0x000000010000a282 drive_machine + 242 (memcached.c:3456) 3 moxi 0x00000001000462c2 event_base_loop + 1154 4 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274) 5 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 6 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 2: 0 libSystem.B.dylib 0x00007fff87c3b272 write + 10 1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230) 2 moxi 0x0000000100001c40 conn_set_state + 160 (memcached.c:668) 3 moxi 0x0000000100026c5d a2b_process_downstream_response + 2397 (cproxy_protocol_a2b.c:764) 4 moxi 0x000000010000ac0c drive_machine + 2684 (memcached.c:1990) 5 moxi 0x00000001000462c2 event_base_loop + 1154 6 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274) 7 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 8 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 3: 0 libSystem.B.dylib 0x00007fff87c3b272 write + 10 1 moxi 0x00000001000314cd log_error_write + 749 (log.c:230) 2 moxi 0x0000000100027a0e cproxy_process_a2b_downstream_nread + 654 (cproxy_protocol_a2b.c:541) 3 moxi 0x000000010000ac0c drive_machine + 2684 (memcached.c:1990) 4 moxi 0x00000001000462c2 event_base_loop + 1154 5 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274) 6 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 7 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 4 Crashed: 0 libSystem.B.dylib 0x00007fff87c770b6 __kill + 10 1 libSystem.B.dylib 0x00007fff87d179f6 abort + 83 2 libSystem.B.dylib 0x00007fff87d049bc __pthread_markcancel + 0 3 moxi 0x000000010001a5b7 cproxy_reserve_downstream + 839 (cproxy.c:926) 4 moxi 0x000000010001a774 cproxy_assign_downstream + 196 (cproxy.c:1785) 5 moxi 0x0000000100021905 cproxy_process_upstream_ascii + 901 (cproxy_protocol_a.c:150) 6 moxi 0x0000000100008083 try_read_command + 835 (memcached.c:3205) 7 moxi 0x000000010000a560 drive_machine + 976 (memcached.c:3536) 8 moxi 0x00000001000462c2 event_base_loop + 1154 9 moxi 0x00000001000128c0 worker_libevent + 128 (thread.c:274) 10 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 11 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 5: 0 libSystem.B.dylib 0x00007fff87c63a6a __semwait_signal + 10 1 libSystem.B.dylib 0x00007fff87c67881 _pthread_cond_wait + 1286 2 moxi 0x0000000100011941 assoc_maintenance_thread + 305 (assoc.c:223) 3 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 4 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 6: 0 libSystem.B.dylib 0x00007fff87c6c932 select$DARWIN_EXTSN + 10 1 libconflate.0.dylib 0x00000001000c2996 Curl_socket_ready + 438 2 libconflate.0.dylib 0x00000001000b67f0 Curl_do_perform + 496 3 libconflate.0.dylib 0x0000000100096719 run_rest_conflate + 801 (rest.c:293) 4 libSystem.B.dylib 0x00007fff87c61fd6 _pthread_start + 331 5 libSystem.B.dylib 0x00007fff87c61e89 thread_start + 13 Thread 4 crashed with X86 Thread State (64-bit): rax: 0x0000000000000000 rbx: 0x0000000100053f05 rcx: 0x0000000100509ac8 rdx: 0x0000000000000000 rdi: 0x00000000000026d0 rsi: 0x0000000000000006 rbp: 0x0000000100509ae0 rsp: 0x0000000100509ac8 r8: 0x0000000000000003 r9: 0x0000000000000000 r10: 0x00007fff87c730fa r11: 0x0000000000000206 r12: 0x000000000000039f r13: 0x000000010005230f r14: 0x0000000100052e36 r15: 0x00007fff70c88790 rip: 0x00007fff87c770b6 rfl: 0x0000000000000206 cr2: 0x0000000102997000 Binary Images: 0x100000000 - 0x100064fef +moxi ??? (???) <64AE645C-76DA-41FB-A00C-C4EB4DDFE5D9> /Users/steveyen/work/membase/dev/voltron.couchbase/build/moxi/moxi 0x100093000 - 0x1000d9fef +libconflate.0.dylib ??? (???) <A2698F74-7010-74E0-F8BF-AB40EA051667> /opt/couchbase/lib/libconflate.0.dylib 0x1000e7000 - 0x1000e9ff7 +libhashkit.0.dylib ??? (???) <EAF5A8E0-FD91-4E2E-07AC-E891EAE0F003> /opt/couchbase/lib/libhashkit.0.dylib 0x1000ed000 - 0x1000f3fef +libvbucket.1.dylib 3.0.0 (compatibility 3.0.0) <2581DCBA-8D50-E90A-ECAF-372830B80C0D> /opt/couchbase/lib/libvbucket.1.dylib 0x1000f9000 - 0x100109fef +libmemcached.6.dylib 7.0.0 (compatibility 7.0.0) <58E35902-BFC2-DF19-5802-7FAC3E201773> /opt/couchbase/lib/libmemcached.6.dylib 0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???) <69130DA3-7CB3-54C8-ABC5-423DECDD2AF7> /usr/lib/dyld 0x7fff813a2000 - 0x7fff813a6ff7 libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /usr/lib/system/libmathCommon.A.dylib 0x7fff85634000 - 0x7fff85645ff7 libz.1.dylib 1.2.3 (compatibility 1.0.0) <97019C74-161A-3488-41EC-A6CA8738418C> /usr/lib/libz.1.dylib 0x7fff87c28000 - 0x7fff87de9fef libSystem.B.dylib 125.2.11 (compatibility 1.0.0) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib 0x7fff8835d000 - 0x7fff8839afff com.apple.LDAPFramework 2.0 (120.1) <54A6769E-D7E2-DBE2-EA61-87B9EA355DA4> /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP 0x7fff88499000 - 0x7fff884abfe7 libsasl2.2.dylib 3.15.0 (compatibility 3.0.0) <76B83C8D-8EFE-4467-0F75-275648AFED97> /usr/lib/libsasl2.2.dylib 0x7fff884ef000 - 0x7fff88510fff libresolv.9.dylib 41.1.0 (compatibility 1.0.0) <9410EC7F-4D24-6740-AFEE-90405750FAD7> /usr/lib/libresolv.9.dylib 0x7fff8a461000 - 0x7fff8a580fe7 libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <14115D29-432B-CF02-6B24-A60CC533A09E> /usr/lib/libcrypto.0.9.8.dylib 0x7fff8a6b3000 - 0x7fff8a6b4ff7 com.apple.TrustEvaluationAgent 1.1 (1) <5952A9FA-BC2B-16EF-91A7-43902A5C07B6> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent 0x7fff8a6f7000 - 0x7fff8a734ff7 libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <F743389F-F25A-A77D-4FCA-D6B01AF2EE6D> /usr/lib/libssl.0.9.8.dylib 0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib

          People

          • Assignee:
            steve Steve Yen
            Reporter:
            keith Keith Batten (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes