Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.1
    • Component/s: moxi
    • Security Level: Public
    • Environment:
      client side moxi version 1.7.0
      cluster version 1.7.1
      moxi command line:
      3 nodes
      centos 5.4 64bit

      Description

      1) Set up 3 node cluster
      2) Start up moxi
      /opt/moxi/bin/moxi -v http://10.1.5.41:8091/pools/default/bucketsStreaming/default,http://10.1.5.42:8091/pools/default/bucketsStreaming/default,http://10.1.5.134:8091/pools/default/bucketsStreaming/default -Z port_listen=11211 -u root -t 16
      3) Start up set/get load against cluster
      4) Start up rebalance in a loop (continually add/remove node)
      5) load 1000 keys to be used for multiget
      6) run ascii multiget in a loop

      moxi: cproxy.c:927: cproxy_reserve_downstream: Assertion `d->timeout_tv.tv_sec == 0' failed.

      Thread 19 (process 24286):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f3baf0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f3baf0, flags=0) at event.c:1558
      #3 0x0000000000408da6 in main (argc=9, argv=0x7fff83c83bf8) at memcached.c:5061

      Thread 18 (process 24287):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f40640, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f40640, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 17 (process 24288):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f44c30, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f44c30, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 16 (process 24289):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f49220, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f49220, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 15 (process 24290):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f4d910, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f4d910, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 14 (process 24291):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f52000, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f52000, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 13 (process 24292):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f566f0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f566f0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 12 (process 24293):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5b150, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5b150, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 11 (process 24294):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5f8d0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5f8d0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 10 (process 24295):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f641c0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f641c0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 9 (process 24296):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f68ab0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f68ab0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 8 (process 24297):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f6d3a0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f6d3a0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 7 (process 24298):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f71c90, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f71c90, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 6 (process 24299):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f76580, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f76580, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 5 (process 24301):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f7fb60, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f7fb60, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 4 (process 24302):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f84850, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f84850, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 3 (process 24303):
      #0 0x0000003e6000ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x0000000000412a50 in assoc_maintenance_thread (arg=<value optimized out>) at assoc.c:220
      #2 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #3 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 2 (process 24304):
      #0 0x0000003e5f4cae46 in poll () from /lib64/libc.so.6
      #1 0x000000000044fc56 in Curl_socket_ready ()
      #2 0x000000000046671b in Curl_do_perform ()
      #3 0x000000000044993b in run_rest_conflate (arg=0x8f8b930) at rest.c:293
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 1 (process 24300):
      #0 0x0000003e5f430265 in raise () from /lib64/libc.so.6
      #1 0x0000003e5f431d10 in abort () from /lib64/libc.so.6
      #2 0x0000003e5f4296e6 in __assert_fail () from /lib64/libc.so.6
      #3 0x000000000041a5ad in cproxy_reserve_downstream (ptd=0x90bbe00) at cproxy.c:927
      #4 0x000000000041a7ae in cproxy_assign_downstream (ptd=0x90bbe00) at cproxy.c:1783
      #5 0x000000000040c86b in drive_machine (c=0x92f8600) at memcached.c:1988
      #6 0x0000000000442321 in event_base_loop (base=0x8f7ae70, flags=0) at event.c:1308
      #7 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #8 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      1. MB-4334_multi_ascii.py
        1 kB
        Steve Yen
      2. MB-4334_rebalance.sh
        0.2 kB
        Steve Yen
      3. MB-4334-moxi-vvv.txt
        720 kB
        Steve Yen

        Activity

        keith Keith Batten (Inactive) created issue -
        steve Steve Yen made changes -
        Field Original Value New Value
        Assignee Steve Yen [ steve ]
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 1.8.1 [ 10295 ]
        steve Steve Yen made changes -
        Attachment MB-4334-moxi-vvv.txt [ 12877 ]
        Attachment MB-4334_multi_ascii.py [ 12878 ]
        Attachment MB-4334_rebalance.sh [ 12879 ]
        dipti Dipti Borkar made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        dipti Dipti Borkar made changes -
        Sprint Status Next Sprint
        Sprint Priority 1
        steve Steve Yen made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        dipti Dipti Borkar made changes -
        Sprint Status Next Sprint Current Sprint
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        steve Steve Yen made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Sprint Status Current Sprint
        steve Steve Yen made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        steve Steve Yen made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        farshid Farshid Ghods (Inactive) made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        farshid Farshid Ghods (Inactive) made changes -
        Labels 1.8.1-release-notes
        farshid Farshid Ghods (Inactive) made changes -
        Status Reopened [ 4 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            steve Steve Yen
            Reporter:
            keith Keith Batten (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes