Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54284

Panic seen in the wild while calling math/rand's Intn(..)

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      The following panic was observed in a 6.5.0 cluster, the exact steps leading to it have not been confirmed. See CBSE-12580 for more details.

      panic: runtime error: index out of range
       
      goroutine 2230682 [running]:
      math/rand.(*rngSource).Uint64(...)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rng.go:249
      math/rand.(*rngSource).Int63(0xc0003ef500, 0x10)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rng.go:234 +0x89
      math/rand.(*Rand).Int63(0xc0004171d0, 0xc304ba7410)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:85 +0x33
      math/rand.(*Rand).Int31(0xc0004171d0, 0x448647)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:99 +0x2b
      math/rand.(*Rand).Int31n(0xc0004171d0, 0xc300000005, 0x31)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:134 +0x4f
      math/rand.(*Rand).Intn(0xc0004171d0, 0x5, 0x20)
          /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/math/rand/rand.go:172 +0x45
      github.com/couchbase/cbft.getRpcClient(0xc009f6c160, 0x20, 0xc30489ab70, 0x10, 0x0, 0x0, 0x0, 0xc30489ab70, 0x10, 0x0, ...)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_util.go:121 +0xbb
      github.com/couchbase/cbft.addGrpcClients(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0xc0037f2a80, 0x18, 0x18, 0x0, 0xc304ae6570, ...)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_client.go:421 +0x39d
      github.com/couchbase/cbft.bleveIndexTargets(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0x4500000000efa201, 0x0, 0xc301b22930, 0x1, 0xc304ae6570, ...)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:2218 +0x3e0
      github.com/couchbase/cbft.bleveIndexAlias(0xc0000a3500, 0xc30489a5c0, 0xe, 0x0, 0x0, 0x1070b01, 0x0, 0xc301b22930, 0xc301b22901, 0xc304ae6570, ...)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/pindex_bleve.go:2045 +0x147
      github.com/couchbase/cbft.(*SearchService).Search(0xc0006fc440, 0xc302331ef0, 0x10755c0, 0xc3047f2f30, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_server.go:194 +0x5f0
      github.com/couchbase/cbft/protobuf._SearchService_Search_Handler(0xe42260, 0xc0006fc440, 0x1074240, 0xc304553780, 0xc00728e5a0, 0x12)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/protobuf/search.pb.go:939 +0x109
      github.com/couchbase/cbft.serverInterceptor(0xe42260, 0xc0006fc440, 0x1074780, 0xc3048f6840, 0xc304553760, 0xf98bd0, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbft/grpc_server.go:342 +0x31c
      google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900, 0xc000371710, 0x1911da0, 0x0, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:1177 +0x488
      google.golang.org/grpc.(*Server).handleStream(0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:1249 +0xcfa
      google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc009a30740, 0xc0004cd980, 0x1077240, 0xc009f5f680, 0xc3047bc900)
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:685 +0x9f
      created by google.golang.org/grpc.(*Server).serveStreams.func1
          /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/google.golang.org/grpc/server.go:683 +0xa1  

      The panic corresponds to this line in our code:

      https://github.com/couchbase/cbft/blob/mad-hatter/grpc_util.go#L121


      We haven't seen this stack trace before, but from some online reading - it seems golang's math/rand's NewSource may not be thread safe (https://pkg.go.dev/math/rand).

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            abhinav Abhi Dangeti added a comment - - edited

            Wayne Siu Lynn Straus This ticket is a clone of MB-53628 which we've addressed in 7.0.5. Seeing that 7.1.3 is in line next for release, we must include this fix. Pending your approval, I can merge the fixes into the neo branches for couchbase/cbft and couchbase/n1fty.

             

            abhinav Abhi Dangeti added a comment - - edited Wayne Siu Lynn Straus This ticket is a clone of MB-53628 which we've addressed in 7.0.5. Seeing that 7.1.3 is in line next for release, we must include this fix. Pending your approval, I can merge the fixes into the neo branches for couchbase/cbft and couchbase/n1fty. https://review.couchbase.org/c/cbft/+/182413 https://review.couchbase.org/c/n1fty/+/182414  

            Build couchbase-server-7.2.0-5013 contains cbft commit b190fad with commit message:
            MB-54284: [BP] rand.NewSource is NOT thread safe

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5013 contains cbft commit b190fad with commit message: MB-54284 : [BP] rand.NewSource is NOT thread safe

            Build couchbase-server-7.2.0-5013 contains n1fty commit 1a61e24 with commit message:
            MB-54284: [BP] rand.NewSource is NOT thread safe

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5013 contains n1fty commit 1a61e24 with commit message: MB-54284 : [BP] rand.NewSource is NOT thread safe

            Build couchbase-server-7.1.3-3471 contains cbft commit b190fad with commit message:
            MB-54284: [BP] rand.NewSource is NOT thread safe

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.3-3471 contains cbft commit b190fad with commit message: MB-54284 : [BP] rand.NewSource is NOT thread safe

            Build couchbase-server-7.1.3-3471 contains n1fty commit 1a61e24 with commit message:
            MB-54284: [BP] rand.NewSource is NOT thread safe

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.3-3471 contains n1fty commit 1a61e24 with commit message: MB-54284 : [BP] rand.NewSource is NOT thread safe

            People

              abhinav Abhi Dangeti
              abhinav Abhi Dangeti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty