Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54284

Panic seen in the wild while calling math/rand's Intn(..)

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      The following panic was observed in a 6.5.0 cluster, the exact steps leading to it have not been confirmed. See CBSE-12580 for more details.

      panic: runtime error: index out of range
      		 
       
      		 
      goroutine 2230682 [running]:
      		 
      math/rand.(*rngSource).Uint64(...)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rng.go:249
      		 
      math/rand.(*rngSource).Int63(0xc0003ef500, 0x10)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rng.go:234 +0x89
      		 
      math/rand.(*Rand).Int63(0xc0004171d0, 0xc304ba7410)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:85 +0x33
      		 
      math/rand.(*Rand).Int31(0xc0004171d0, 0x448647)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:99 +0x2b
      		 
      math/rand.(*Rand).Int31n(0xc0004171d0, 0xc300000005, 0x31)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:134 +0x4f
      		 
      math/rand.(*Rand).Intn(0xc0004171d0, 0x5, 0x20)
      		 
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:172 +0x45
      		 
      github.com/couchbase/cbft.getRpcClient(0xc009f6c160, 0x20, 0xc30489ab70, 0x10, 0x0, 0x0, 0x0, 0xc30489ab70, 0x10, 0x0, ...)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_util.go:121 +0xbb
      		 
      github.com/couchbase/cbft.addGrpcClients(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0xc0037f2a80, 0x18, 0x18, 0x0, 0xc304ae6570, ...)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_client.go:421 +0x39d
      		 
      github.com/couchbase/cbft.bleveIndexTargets(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0x4500000000efa201, 0x0, 0xc301b22930, 0x1, 0xc304ae6570, ...)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:2218 +0x3e0
      		 
      github.com/couchbase/cbft.bleveIndexAlias(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0x1070b01, 0x0, 0xc301b22930, 0xc301b22901, 0xc304ae6570, ...)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:2045 +0x147
      		 
      github.com/couchbase/cbft.(*SearchService).Search(0xc0006fc440, 0xc302331ef0, 0x10755c0, 0xc3047f2f30, 0x0, 0x0)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_server.go:194 +0x5f0
      		 
      github.com/couchbase/cbft/protobuf._SearchService_Search_Handler(0xe42260, 0xc0006fc440, 0x1074240, 0xc304553780, 0xc00728e5a0, 0x12)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/protobuf/search.pb.go:939 +0x109
      		 
      github.com/couchbase/cbft.serverInterceptor(0xe42260, 0xc0006fc440, 0x1074780, 0xc3048f6840, 0xc304553760, 0xf98bd0, 0x0, 0x0)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_server.go:342 +0x31c
      		 
      google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900, 0xc000371710, 0x1911da0, 0x0, 0x0, 0x0)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:1177 +0x488
      		 
      google.golang.org/grpc.(*Server).handleStream(0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900, 0x0)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:1249 +0xcfa
      		 
      google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc009a30740, 0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900)
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:685 +0x9f
      		 
      created by google.golang.org/grpc.(*Server).serveStreams.func1
      		 
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:683 +0xa1 

      The panic corresponds to this line in our code:

      https://github.com/couchbase/cbft/blob/mad-hatter/grpc_util.go#L121

      We haven't seen this stack trace before, but from some online reading - it seems golang's math/rand's NewSource may not be thread safe (https://pkg.go.dev/math/rand).

      Attachments

        For Gerrit Dashboard: MB-54284
        # Subject Branch Project Status CR V
        182413,2 MB-54284: [BP] rand.NewSource is NOT thread safe neo cbft Status: MERGED +2 +1
        182414,2 MB-54284: [BP] rand.NewSource is NOT thread safe neo n1fty Status: MERGED +2 +1

        Activity

          People

            abhinav Abhi Dangeti
            abhinav Abhi Dangeti
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty