Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31432

cbexport may hang when export a bucket with many invalid json documents

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 6.5.0
    • 6.0.0, 6.5.0
    • tools
    • None
    • Centos 7
    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

      Install 6.0.0-1661 on 4 centos 7 servers

      Create default bucket with 256 MB quota

      Load 1000 json docs to default bucket

      Load any data (not valid json) to bucket until active resident ratio reach around 30%

      Run cbexport to export data out of bucket.  There is a chance that cbexport hang forever

      Kill that cbexport process and run again, it passed.

       

      Here is the backtrack from the hang process

      (gdb) attach 29277
      Attaching to process 29277
      Reading symbols from /opt/couchbase/bin/cbexport...Reading symbols from /usr/lib/debug/opt/couchbase/bin/cbexport.debug...done.
      warning: Missing auto-load scripts referenced in section .debug_gdb_scripts
      of file /usr/lib/debug/opt/couchbase/bin/cbexport.debug
      Use `info auto-load python [REGEXP]' to list them.
      done.
      Reading symbols from /opt/couchbase/bin/../lib/libforestdb.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libforestdb.so.debug...done.
      done.
      Loaded symbols for /opt/couchbase/bin/../lib/libforestdb.so
      Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
      done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
      done.
      [New LWP 29292]
      [New LWP 29287]
      [New LWP 29286]
      [New LWP 29285]
      [New LWP 29284]
      [New LWP 29283]
      [New LWP 29282]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
      done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /opt/couchbase/bin/../lib/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done.
      done.
      Loaded symbols for /opt/couchbase/bin/../lib/../lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/bin/../lib/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done.
      done.
      Loaded symbols for /opt/couchbase/bin/../lib/../lib/libjemalloc.so.2
      Reading symbols from /lib64/librt.so.1...Reading symbols from /usr/lib/debug/usr/lib64/librt-2.17.so.debug...done.
      done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /opt/couchbase/bin/../lib/../lib/libstdc++.so.6...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libstdc++.so.debug...done.
      done.
      Loaded symbols for /opt/couchbase/bin/../lib/../lib/libstdc++.so.6
      Reading symbols from /lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.17.so.debug...done.
      done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /opt/couchbase/bin/../lib/../lib/libgcc_s.so.1...done.
      Loaded symbols for /opt/couchbase/bin/../lib/../lib/libgcc_s.so.1
      Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
      done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      426	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s: No such file or directory.
      (gdb)  thread apply all btThread 8 (Thread 0x7f5877dff700 (LWP 29282)):
      #0  runtime.usleep () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:98
      #1  0x0000000000441b03 in runtime.sysmon () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:3793
      #2  0x000000000043acee in runtime.mstart1 () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1179
      #3  0x000000000043abc4 in runtime.mstart () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1149
      #4  0x000000000082522c in crosscall_amd64 () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/cgo/gcc_amd64.S:35
      #5  0x00007f5877dff700 in ?? ()
      #6  0x0000000000000000 in ?? ()Thread 7 (Thread 0x7f58775fe700 (LWP 29283)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433e47 in runtime.futexsleep (addr=0xd345d8 <runtime.timers+24>, val=0, ns=101891854) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:62
      #2  0x000000000041b210 in runtime.notetsleep_internal (n=0xd345d8 <runtime.timers+24>, ns=101891854, ~r2=88) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:174
      #3  0x000000000041b36a in runtime.notetsleepg (n=0xd345d8 <runtime.timers+24>, ns=101891854, ~r2=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:206
      #4  0x0000000000453687 in runtime.timerproc () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/time.go:209
      #5  0x00000000004621b1 in runtime.goexit () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:2197
      #6  0x0000000000000000 in ?? ()Thread 6 (Thread 0x7f5876dfd700 (LWP 29284)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433dd2 in runtime.futexsleep (addr=0xc420026d10, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:45
      #2  0x000000000041b0eb in runtime.notesleep (n=0xc420026d10) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:145
      #3  0x000000000043bc1d in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1650
      #4  0x000000000043ca34 in runtime.findrunnable (gp#10=0xc42001b900, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2102
      #5  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #6  0x000000000043d8cb in runtime.park_m (gp=0xc4201aa000) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #7  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #8  0x000000c420018000 in ?? ()
      #9  0x0000000000000000 in ?? ()Thread 5 (Thread 0x7f58765fc700 (LWP 29285)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433dd2 in runtime.futexsleep (addr=0xc4200b7510, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:45
      #2  0x000000000041b0eb in runtime.notesleep (n=0xc4200b7510) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:145
      #3  0x000000000043bc1d in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1650
      #4  0x000000000043ca34 in runtime.findrunnable (gp#10=0xc42001a600, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2102
      #5  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #6  0x000000000043d8cb in runtime.park_m (gp=0xc4242e3040) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #7  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #8  0x000000c420019300 in ?? ()
      #9  0x0000000000000000 in ?? ()Thread 4 (Thread 0x7f5875dfb700 (LWP 29286)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433dd2 in runtime.futexsleep (addr=0xc4241c8110, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:45
      #2  0x000000000041b0eb in runtime.notesleep (n=0xc4241c8110) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:145
      #3  0x000000000043bc1d in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1650
      #4  0x000000000043ca34 in runtime.findrunnable (gp#10=0xc42001a600, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2102
      #5  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #6  0x000000000043d8cb in runtime.park_m (gp=0xc4243aa1a0) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #7  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #8  0x000000c42001a600 in ?? ()
      #9  0x0000000000000000 in ?? ()Thread 3 (Thread 0x7f58755fa700 (LWP 29287)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433dd2 in runtime.futexsleep (addr=0xc4241d2110, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:45
      #2  0x000000000041b0eb in runtime.notesleep (n=0xc4241d2110) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:145
      #3  0x000000000043bc1d in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1650
      #4  0x000000000043ca34 in runtime.findrunnable (gp#10=0xc42001b900, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2102
      #5  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #6  0x000000000043d8cb in runtime.park_m (gp=0xc4242e31e0) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #7  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #8  0x000000c42001b900 in ?? ()
      #9  0x0000000000000000 in ?? ()---Type <return> to continue, or q <return> to quit---
      Thread 2 (Thread 0x7f5874d09700 (LWP 29292)):
      #0  runtime.epollwait () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:563
      #1  0x0000000000433bb1 in runtime.netpoll (block=true, ~r1=0x1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/netpoll_epoll.go:67
      #2  0x000000000043ca6f in runtime.findrunnable (gp#10=0xc420018000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2084
      #3  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #4  0x000000000043d8cb in runtime.park_m (gp=0xc4242b0340) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #5  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #6  0x000000c420018000 in ?? ()
      #7  0x0000000000000000 in ?? ()Thread 1 (Thread 0x7f5879d957c0 (LWP 29277)):
      #0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/sys_linux_amd64.s:426
      #1  0x0000000000433dd2 in runtime.futexsleep (addr=0xd352b0 <runtime.m0+272>, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/os_linux.go:45
      #2  0x000000000041b0eb in runtime.notesleep (n=0xd352b0 <runtime.m0+272>) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/lock_futex.go:145
      #3  0x000000000043bc1d in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1650
      #4  0x000000000043ca34 in runtime.findrunnable (gp#10=0xc420019300, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2102
      #5  0x000000000043d5cc in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2222
      #6  0x000000000043d8cb in runtime.park_m (gp=0xc4201aa000) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:2285
      #7  0x000000000045f5db in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:269
      #8  0x0000000000d34e00 in runtime.work ()
      #9  0x00007ffc9d874b80 in ?? ()
      #10 0x0000000000d34e40 in runtime.work ()
      #11 0x00007ffc9d874b70 in ?? ()
      #12 0x000000000043abc4 in runtime.mstart () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/proc.go:1149
      #13 0x000000000045f449 in runtime.rt0_go () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/asm_amd64.s:169
      #14 0x0000000000000010 in ?? ()
      #15 0x00007ffc9d874c88 in ?? ()
      #16 0x0000000000000010 in ?? ()
      #17 0x00007ffc9d874c88 in ?? ()
      #18 0x0000000000000000 in ?? () 

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pvarley Patrick Varley (Inactive)
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty