Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-63220

Observed a panic in indexer.log

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus, Cypher
    • Morpheus, Cypher
    • secondary-index
    • Enterprise Edition 7.7.0 build 1046
    • Untriaged
    • 0
    • No

    Description

      Observed the panic in indexer.log

       

      2024-08-20T03:33:33.587-04:00 [Error] Indexer::persistCodebookToDisk Error observed while initializing codebook dir for instId: 17496810518424195641, partnId: 4, sliceId: 0
      2024-08-20T03:33:33.587-04:00 [Error] Indexer::initiateTraining error observed while persisting codebook for instId: 17496810518424195641, partnId: 4, err: mkdir /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_docloadera12184bdd9304a02afae
      99ecfe82b945multiScalarOneVector_2_17496810518424195641_4.index/codebook: no such file or directory
      2024-08-20T03:33:33.587-04:00 [Info] Indexer::removeCodebookDir removal successful for instId: 17496810518424195641, partnId: 5, sliceId: 0
      2024-08-20T03:33:33.587-04:00 [Info] Indexer::removeCodebookDir removal successful for instId: 17496810518424195641, partnId: 6, sliceId: 0
      2024-08-20T03:33:33.587-04:00 [Info] Indexer::removeCodebookDir removal successful for instId: 17496810518424195641, partnId: 3, sliceId: 0
      2024-08-20T03:33:33.587-04:00 [Info] Indexer::removeCodebookDir removal successful for instId: 17496810518424195641, partnId: 4, sliceId: 0
      panic: assignment to entry in nil map
       
       
      goroutine 1 [running]:
      github.com/couchbase/indexing/secondary/indexer.(*indexer).handleIndexTrainingDone(0x40004fa808, {0x1ff5840?, 0x4007d62c40?})
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:13582 +0x89c
      github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0x40004fa808, {0x1ff5840, 0x4007d62c40})
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1777 +0xda0
      github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40004fa808)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1289 +0x268
      github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000464090)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:641 +0x21cc
      main.main()
              goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:207 +0x1004
      2024-08-20T03:33:33.844-04:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=ffb6683a48973399f50d3dda6ea3a76b -isEnterprise=true --deploymentModel=default]
      2024-08-20T03:33:33.844-04:00 [Info] Indexer DeploymentModel is set to: default 

       

      Also, observe panic in babysitter.log

      error_logger:error,2024-08-16T06:18:38.465-04:00,babysitter_of_ns_1@cb.local:<0.251.0>:ale_error_logger_handler:do_log:101]
      =========================ERROR REPORT=========================
      ** Generic server <0.251.0> terminating 
      ** Last message in was {<0.257.0>,{exit_status,2}}
      ** When Server state == {state,<0.257.0>,2463,
                               {index,"/opt/couchbase/bin/indexer",
                                ["-adminPort=9100","-scanPort=9101",
                                 "-httpPort=9102","-streamInitPort=9103",
                                 "-streamCatchupPort=9104","-streamMaintPort=9105",
                                 "--httpsPort=19102",
                                 "--certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem",
                                 "--keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem",
                                 "--caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem",
                                 "--clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem",
                                 "--clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem",
                                 "-ipv4=required","-ipv6=optional",
                                 "-cluster=127.0.0.1:8091",
                                 "-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i",
                                 "-diagDir=/opt/couchbase/var/lib/couchbase/crash",
                                 "-logDir=/opt/couchbase/var/lib/couchbase/logs",
                                 "-nodeUUID=ffb6683a48973399f50d3dda6ea3a76b",
                                 "-isEnterprise=true","--deploymentModel=default"],
                                [via_goport,exit_status,stderr_to_stdout,
                                 {env,
                                  [{"GOTRACEBACK","single"},
                                   {"GOMAXPROCS","8"},
                                   {"CBAUTH_REVRPC_URL",
                                    {dynamic,
                                     {ns_ports_setup,
                                      remote_build_cbauth_revrpc_url,
                                      [index]}}}]}]},
                               {ringbuffer,2048,1024,
                                {[{<<"...)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1172 +0x88\n"...>>,
                                   642}],
                                 [{<<"panic: runtime error: slice bounds out of range [:4] with capacity 0 [recovered]\n\tpanic: runtime error: slice bounds out of range [:4] with capa"...>>,
                                   1406}]}},
                               index,undefined,[],0}
      ** Reason for termination ==
      ** {abnormal,2}
       

       
      https://cb-engineering.s3.amazonaws.com/vector-index/collectinfo-2024-08-20T074511-ns_1%40192.168.80.3.zip
      https://cb-engineering.s3.amazonaws.com/vector-index/collectinfo-2024-08-20T074511-ns_1%40192.168.80.4.zip
      https://cb-engineering.s3.amazonaws.com/vector-index/collectinfo-2024-08-20T074511-ns_1%40192.168.80.5.zip
       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dhananjay.kshirsagar Dhananjay Kshirsagar
            hemant.rajput Hemant Rajput
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty