Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47988

[BP 7.0.2 MB-47475] - Prevent Indexer Crash with PlasmaDiag invalid DB instance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Major
    • 7.1.0
    • None
    • storage-engine
    • Untriaged
    • 1
    • Unknown

    Description

      Today, we pass db instance which are basically golang pointers as argument for plasmaDiag REST API calls. If we provide garbage as instance ids, dereferencing can either lead to panics or unhandled exceptions such as the one below. For the latter, is not a recoverable error in golang and results in crashing the indexer.
       
         In addition from security point of view, it is not advisable to expose internal memory addresses through any REST APIs.
       
      indexer.log:
       
      2021-07-15T18:35:00.300-07:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 8] [c62eca2248e32c85977ca9f23b06526d 401ed77c7ecd37ba800de7ef65c10cc4 4af441577ad164362dd80795081f7ab1 47679e6d345699b6db3e938577573f32] true []}
      2021-07-15T18:35:00.303-07:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 8]
      2021-07-15T18:35:11.696-07:00 [Info] ServiceMgr::rebalanceJanitor Running Periodic Cleanup
      unexpected fault address 0x6967
      fatal error: fault
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x6967 pc=0x491846b]
       
      goroutine 4338 [running]:
      runtime.throw(0x4ea51a3, 0x5)
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:1117 +0x72 fp=0xc00561ebb8 sp=0xc00561eb88 pc=0x403bbb2
      runtime.sigpanic()
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/signal_unix.go:741 +0x276 fp=0xc00561ebf0 sp=0xc00561ebb8 pc=0x4053216
      github.com/couchbase/plasma.(*Plasma).getStats(0x63df, 0x4ec8f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/plasma.go:851 +0xab fp=0xc00561eed8 sp=0xc00561ebf0 pc=0x491846b
      github.com/couchbase/plasma.(*Plasma).GetStats(...)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/plasma.go:834
      github.com/couchbase/plasma.(*diag).Command(0x5940568, 0xc0062f6bc8, 0x5, 0xc005f7ed00, 0xc005f7ecc0, 0x1, 0x4)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/diag.go:164 +0xe4d fp=0xc00561f9d0 sp=0xc00561eed8 pc=0x48e284d
      github.com/couchbase/plasma.(*diag).HandleHttp(0x5940568, 0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/diag.go:362 +0x19e fp=0xc00561fa58 sp=0xc00561f9d0 pc=0x48e327e
      github.com/couchbase/indexing/secondary/indexer.(*settingsManager).handlePlasmaDiag(0xc0001541e0, 0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:240 +0xfa fp=0xc00561fab8 sp=0xc00561fa58 pc=0x4b46e1a
      github.com/couchbase/indexing/secondary/indexer.(*settingsManager).handlePlasmaDiag-fm(0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:230 +0x51 fp=0xc00561fae8 sp=0xc00561fab8 pc=0x4beceb1
      net/http.HandlerFunc.ServeHTTP(0xc000629730, 0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/http/server.go:2069 +0x44 fp=0xc00561fb10 sp=0xc00561fae8 pc=0x42ce484
      net/http.(*ServeMux).ServeHTTP(0xc00061f1c0, 0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/http/server.go:2448 +0x1ad fp=0xc00561fb70 sp=0xc00561fb10 pc=0x42d030d
      net/http.serverHandler.ServeHTTP(0xc0006627e0, 0x50596c0, 0xc000277b20, 0xc004a72a00)
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/http/server.go:2887 +0xa3 fp=0xc00561fba0 sp=0xc00561fb70 pc=0x42d1a43
      net/http.(*conn).serve(0xc005993680, 0x5063860, 0xc005f7eb80)
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/http/server.go:1952 +0x8cd fp=0xc00561ffc8 sp=0xc00561fba0 pc=0x42ccf6d
      runtime.goexit()
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00561ffd0 sp=0xc00561ffc8 pc=0x4075281
      created by net/http.(*Server).Serve
      /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/http/server.go:3013 +0x39b
       
      client side:
      python3 diag_client.py --port 9102 --stats 1

      default/#primary/Mainstore#3083721504090702076:0

      Traceback (most recent call last):

        File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen

          httplib_response = self._make_request(

        File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request

          six.raise_from(e, None)

        File "<string>", line 3, in raise_from

        File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request

          httplib_response = conn.getresponse()

        File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1347, in getresponse

          response.begin()

        File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 307, in begin

          version, status, reason = self._read_status()

        File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 276, in _read_status

          raise RemoteDisconnected("Remote end closed connection without"

      http.client.RemoteDisconnected: Remote end closed connection without response
       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              saptarshi.sen Saptarshi Sen
              saptarshi.sen Saptarshi Sen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty