Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19977

indexer exited during jemalloc_free() from fdb_rollback

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 4.5.1
    • 4.5.0
    • forestdb
    • None
    • Untriaged
    • Unknown
    • ForestDB: Oct 17 - Nov 4

    Description

      4.5 GA, 12 days up, indexer crashed 3 hours after encountering MB-19975.

      Unfortunately logs rolled over before I was able to back them up. From ui:

      ervice 'indexer' exited with status 1. Restarting. Messages: runtime.goexit()
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8a7472fb0 sp=0xc8a7472fa8
      created by net/http.(*Transport).dialConn
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/net/http/transport.go:853 +0x10a6
      [goport] 2016/06/20 11:21:48 /opt/couchbase/bin/indexer terminated: signal: aborted (core dumped)
       
      Service 'indexer' exited with status 1. Restarting. Messages: 2016-06-20T11:26:55.528-07:00 [Info] SCAN##294930 RESPONSE status:(error = Indexer In Bootstrap State. Please retry the request later.), requestId: c4cc7fa2-54e6-45aa-a7b5-b9aae7f807dd
      2016-06-20T11:26:55.528-07:00 [Info] SCAN##294931 REQUEST defnId:3960761121304801551, index:/, type:scan, span:range (%!s(<nil>),%!s(<nil>) incl:both), limit:9223372036854775807, requestId:6766cd7d-5eda-4266-8c1b-6b6a38ad496d
      2016-06-20T11:26:55.528-07:00 [Info] SCAN##294931 RESPONSE status:(error = Indexer In Bootstrap State. Please retry the request later.), requestId: 6766cd7d-5eda-4266-8c1b-6b6a38ad496d
      2016-06-20T11:26:55.531-07:00 [Fatal] Indexer::validateIndexInstMap Bucket CUSTOMER not ready even after max retries. Restarting indexer.
      [goport] 2016/06/20 11:26:55 /opt/couchbase/bin/indexer terminated: exit status 1
      

      Looks like forestdb was doing a rollback at time of exiting.
      172.23.105.60:/data/core.indexer.12929

      Thread 12 (Thread 0x7f00357fa700 (LWP 12981)):
      #0  je_free (ptr=0x7efd07667e60) at src/jemalloc.c:1837
      #1  0x00007f0335eb1223 in _flush_dirty_blocks (fname_item=0x7efd073b2280, flush_all=true, immutables_only=false, sync=true)
          at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/blockcache.cc:620
      #2  0x00007f0335ec982a in filemgr_commit_bid (file=0x7efd08a0ef00, bid=18446744073709551615, bmp_revnum=5, sync=<optimized out>, log_callback=0x7f0084a834b0)
          at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/filemgr.cc:2321
      #3  0x00007f0335eea8ef in _fdb_kvs_remove (fhandle=fhandle@entry=0x7f0084a18580, kvs_name=<optimized out>, rollback_recreate=true)
          at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/kv_instance.cc:2058
      #4  0x00007f0335eecf4f in fdb_kvs_rollback (handle_ptr=0xc82509d668, seqnum=0) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/kv_instance.cc:2168
      #5  0x00007f0335ed699d in fdb_rollback (handle_ptr=0xc82509d668, seqnum=0) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:1194
      #6  0x0000000000c8a257 in _cgo_ea7381d1a5e0_Cfunc_fdb_rollback (v=0xcafe84f820)
          at /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/commit.go:50
      #7  0x0000000000464010 in runtime.asmcgocall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:582
      #8  0x0000000000000008 in ?? ()
      #9  0x000000cafe84f7a0 in ?? ()
      #10 0x00000000004073ba in runtime.cgocall (fn=0xc820022a00, arg=0x0, ~r2=629030016) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:110
      #11 0x0000000000436a30 in runtime.startTheWorldWithSema () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:983
      #12 0x000000c820022a00 in ?? ()
      #13 0x0000000000000000 in ?? ()
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              tai.tran Tai Tran (Inactive)
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty