Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54204

Service 'index' exited with status 2. couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone while deleting a bucket during index building.

    XMLWordPrintable

Details

    Description

      Create 20 databases and start deleting them while some of them are build and other building.

      Service 'index' exited with status 2. Restarting. Messages:
      github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone(0x4000183880?, {0x1a0ca58?, 0x40005a9f80?})
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7708 +0x8ac
      github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0x4000183880, {0x1a0ca58, 0x40005a9f80})
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1493 +0xf28
      github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000183880)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1123 +0x274
      github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x272b270?)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:549 +0x1f74
      main.main()
      goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:206 +0xf90
      

      The 2 indexer nodes in cluster are:

      • 83xl37vwyjttp7pf.ker7pmmsnib4vzc.sandbox.nonprod-project-avengers.com
      • azjiuetv5hqlnutp.ker7pmmsnib4vzc.sandbox.nonprod-project-avengers.com

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jliang John Liang added a comment - - edited

            Ritesh Agarwal It is the same problem as you filed today.

            2022-10-19T22:09:28.047+00:00 WARN REGU.throttle.(*throttleimpl).ensureThrottleInfoFromThrottleCtx() at throttleimpl.go:225 [id 86549] [index] refusing to create throttleInfo on unknown bucket volumetestbucket-13-w8f5gy 
            2022-10-19T22:09:28.053+00:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(SJCSEA000>>11->>12->>12->>12->>12->>12->>13->>14->>14->>14->>15->>16->>16->>16-)</ud> docid = <ud>(route_6800)</ud> Index def_route_src_dst_day, Bucket volumetestbucket-13-w8f5gy, IndexInstId 9485841829590938461, PartitionId 0
            2022-10-19T22:09:28.107+00:00 [Fatal] goroutine 82203 [running]:
            github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175
            github.com/couchbase/indexing/secondary/logging.StackTraceAll()
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x4c
            github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1()
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:741 +0x260
            panic({0x13a2360, 0x40000994a0})
            	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.18.5/go/src/runtime/panic.go:838 +0x20c
            github.com/couchbase/regulator/impl.(*transaction).flushLOCKED(0x403317c1c0)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/regulator/impl/transaction.go:185 +0x29c
            github.com/couchbase/regulator/impl.(*transaction).Commit(0x403317c1c0)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/regulator/impl/transaction.go:87 +0xd8
            github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex.func1()
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1102 +0x44
            github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex(0x4005660100, {0x402254e780, 0x7b, 0x80}, {0x40144de540, 0xa, 0x10}, 0x6, 0x0, 0x403509a280)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1283 +0x1934
            github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0x4005660100, {0x402254e780?, 0x272ee80?, 0x0?}, {0x40144de540?, 0x100000001634201?, 0x400713ff80?}, 0x16?, 0x16?, 0x400fc20568?)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:880 +0x70
            github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0x4005660100, 0x6)
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:755 +0x2f4
            created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
            	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3450 +0x3
            

            jliang John Liang added a comment - - edited Ritesh Agarwal It is the same problem as you filed today. 2022-10-19T22:09:28.047+00:00 WARN REGU.throttle.(*throttleimpl).ensureThrottleInfoFromThrottleCtx() at throttleimpl.go:225 [id 86549] [index] refusing to create throttleInfo on unknown bucket volumetestbucket-13-w8f5gy 2022-10-19T22:09:28.053+00:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(SJCSEA000>>11->>12->>12->>12->>12->>12->>13->>14->>14->>14->>15->>16->>16->>16-)</ud> docid = <ud>(route_6800)</ud> Index def_route_src_dst_day, Bucket volumetestbucket-13-w8f5gy, IndexInstId 9485841829590938461, PartitionId 0 2022-10-19T22:09:28.107+00:00 [Fatal] goroutine 82203 [running]: github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175 github.com/couchbase/indexing/secondary/logging.StackTraceAll() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x4c github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:741 +0x260 panic({0x13a2360, 0x40000994a0}) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.18.5/go/src/runtime/panic.go:838 +0x20c github.com/couchbase/regulator/impl.(*transaction).flushLOCKED(0x403317c1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/regulator/impl/transaction.go:185 +0x29c github.com/couchbase/regulator/impl.(*transaction).Commit(0x403317c1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/regulator/impl/transaction.go:87 +0xd8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex.func1() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1102 +0x44 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex(0x4005660100, {0x402254e780, 0x7b, 0x80}, {0x40144de540, 0xa, 0x10}, 0x6, 0x0, 0x403509a280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1283 +0x1934 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0x4005660100, {0x402254e780?, 0x272ee80?, 0x0?}, {0x40144de540?, 0x100000001634201?, 0x400713ff80?}, 0x16?, 0x16?, 0x400fc20568?) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:880 +0x70 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0x4005660100, 0x6) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:755 +0x2f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3450 +0x3

            oops, thanks for a quick check John Liang. Closing it.

            ritesh.agarwal Ritesh Agarwal added a comment - oops, thanks for a quick check John Liang . Closing it.

            People

              amit.kulkarni Amit Kulkarni
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty