Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4334

moxi assert during rebalance/multiget

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 1.8.1
    • None
    • moxi
    • Security Level: Public
    • client side moxi version 1.7.0
      cluster version 1.7.1
      moxi command line:
      3 nodes
      centos 5.4 64bit

    Description

      1) Set up 3 node cluster
      2) Start up moxi
      /opt/moxi/bin/moxi -v http://10.1.5.41:8091/pools/default/bucketsStreaming/default,http://10.1.5.42:8091/pools/default/bucketsStreaming/default,http://10.1.5.134:8091/pools/default/bucketsStreaming/default -Z port_listen=11211 -u root -t 16
      3) Start up set/get load against cluster
      4) Start up rebalance in a loop (continually add/remove node)
      5) load 1000 keys to be used for multiget
      6) run ascii multiget in a loop

      moxi: cproxy.c:927: cproxy_reserve_downstream: Assertion `d->timeout_tv.tv_sec == 0' failed.

      Thread 19 (process 24286):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f3baf0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f3baf0, flags=0) at event.c:1558
      #3 0x0000000000408da6 in main (argc=9, argv=0x7fff83c83bf8) at memcached.c:5061

      Thread 18 (process 24287):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f40640, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f40640, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 17 (process 24288):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f44c30, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f44c30, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 16 (process 24289):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f49220, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f49220, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 15 (process 24290):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f4d910, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f4d910, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 14 (process 24291):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f52000, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f52000, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 13 (process 24292):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f566f0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f566f0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 12 (process 24293):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5b150, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5b150, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 11 (process 24294):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5f8d0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f5f8d0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 10 (process 24295):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f641c0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f641c0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 9 (process 24296):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f68ab0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f68ab0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 8 (process 24297):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f6d3a0, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f6d3a0, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 7 (process 24298):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f71c90, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f71c90, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 6 (process 24299):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f76580, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f76580, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 5 (process 24301):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f7fb60, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f7fb60, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 4 (process 24302):
      #0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
      #1 0x0000000000447ab4 in epoll_dispatch (base=0x8f84850, tv=<value optimized out>) at epoll.c:404
      #2 0x0000000000441f95 in event_base_loop (base=0x8f84850, flags=0) at event.c:1558
      #3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 3 (process 24303):
      #0 0x0000003e6000ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x0000000000412a50 in assoc_maintenance_thread (arg=<value optimized out>) at assoc.c:220
      #2 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #3 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 2 (process 24304):
      #0 0x0000003e5f4cae46 in poll () from /lib64/libc.so.6
      #1 0x000000000044fc56 in Curl_socket_ready ()
      #2 0x000000000046671b in Curl_do_perform ()
      #3 0x000000000044993b in run_rest_conflate (arg=0x8f8b930) at rest.c:293
      #4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Thread 1 (process 24300):
      #0 0x0000003e5f430265 in raise () from /lib64/libc.so.6
      #1 0x0000003e5f431d10 in abort () from /lib64/libc.so.6
      #2 0x0000003e5f4296e6 in __assert_fail () from /lib64/libc.so.6
      #3 0x000000000041a5ad in cproxy_reserve_downstream (ptd=0x90bbe00) at cproxy.c:927
      #4 0x000000000041a7ae in cproxy_assign_downstream (ptd=0x90bbe00) at cproxy.c:1783
      #5 0x000000000040c86b in drive_machine (c=0x92f8600) at memcached.c:1988
      #6 0x0000000000442321 in event_base_loop (base=0x8f7ae70, flags=0) at event.c:1308
      #7 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
      #8 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
      #9 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6

      Attachments

        For Gerrit Dashboard: MB-4334
        # Subject Branch Project Status CR V

        Activity

          People

            steve Steve Yen
            keith Keith Batten (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty