Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54466

FTS crashed during index creation

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Elixir
    • Elixir
    • fts
    • 7.5.0-3163

    Description

      1. create 75 databases
      2. create 5 and 10 fts indexes on databases alternatively.
      3. FTS crashed during the index creation. RAM is at >95% just during the indexes creation.

        Service 'fts' exited with status 2. Restarting. Messages:
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/server.go:786 +0x38
        created by google.golang.org/grpc.(*Server).handleRawConn
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/server.go:785 +0x3fc
         
        goroutine 5599002 [select]:
        google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x476d2bf4a0, 0x1)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/controlbuf.go:395 +0x15c
        google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x46e3aeb9e0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/controlbuf.go:515 +0x78
        google.golang.org/grpc/internal/transport.newHTTP2Server.func2()
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http2_server.go:291 +0xdc
        created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.31.0/internal/transport/http2_server.go:288 +0x115c
        

      On node o52smyma8clav-lj.8eh8ignk6evjoni9.sandbox.nonprod-project-avengers.com system memory use is 97.14% of total available memory, above the critical threshold of 95%.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ritesh Agarwal I see a new build's available with that fix^. When you're ready - can you let us know how we do.

            abhinav Abhi Dangeti added a comment - Ritesh Agarwal I see a new build's available with that fix^. When you're ready - can you let us know how we do.
            shelby.ramsey Shelby Ramsey added a comment - - edited

            Ritesh Agarwal what were the index creation statements for this test?

            shelby.ramsey Shelby Ramsey added a comment - - edited Ritesh Agarwal what were the index creation statements for this test?

            Hello Shelby Ramsey, the fts index definition looks something like below:

                def get_fts_idx_template(self):
                    fts_idx_template = {
                        "type": "fulltext-index",
                        "name": "fts-index",
                        "sourceType": "gocbcore",
                        "sourceName": "default",
                        "planParams": {
                            "maxPartitionsPerPIndex": 1024,
                            "indexPartitions": 1,
                            "numReplicas": 1
                         },
                        "params": {
                            "doc_config": {
                                "docid_prefix_delim": "",
                                "docid_regexp": "",
                                "mode": "scope.collection.type_field",
                                "type_field": "type"
                                },
                            "mapping": {
                                "analysis": {},
                                "default_analyzer": "standard",
                                "default_datetime_parser": "dateTimeOptional",
                                "default_field": "_all",
                                "default_mapping": {
                                    "dynamic": True,
                                    "enabled": False
                                    },
                                "default_type": "_default",
                                "docvalues_dynamic": False,
                                "index_dynamic": True,
                                "store_dynamic": False,
                                "type_field": "_type",
                                "types": {scope.collection: "dynamic": True, "enabled": True}
                                },
                            "store": {
                                "indexType": "scorch",
                                "segmentVersion": 15
                                }
                            },
                        "sourceParams": {}
                       }
                    return fts_idx_template
            

            I think it indexes all fields in the collection on which it is created. As of now i create the same indexes across buckets/collections. I will work on indexes improvement/variety soon...

            ritesh.agarwal Ritesh Agarwal added a comment - Hello Shelby Ramsey , the fts index definition looks something like below: def get_fts_idx_template(self): fts_idx_template = { "type" : "fulltext-index" , "name" : "fts-index" , "sourceType" : "gocbcore" , "sourceName" : "default" , "planParams" : { "maxPartitionsPerPIndex" : 1024 , "indexPartitions" : 1 , "numReplicas" : 1 }, "params" : { "doc_config" : { "docid_prefix_delim" : "" , "docid_regexp" : "" , "mode" : "scope.collection.type_field" , "type_field" : "type" }, "mapping" : { "analysis" : {}, "default_analyzer" : "standard" , "default_datetime_parser" : "dateTimeOptional" , "default_field" : "_all" , "default_mapping" : { "dynamic" : True, "enabled" : False }, "default_type" : "_default" , "docvalues_dynamic" : False, "index_dynamic" : True, "store_dynamic" : False, "type_field" : "_type" , "types" : {scope.collection: "dynamic" : True, "enabled" : True} }, "store" : { "indexType" : "scorch" , "segmentVersion" : 15 } }, "sourceParams" : {} } return fts_idx_template I think it indexes all fields in the collection on which it is created. As of now i create the same indexes across buckets/collections. I will work on indexes improvement/variety soon...

            Hello Abhi Dangeti, I am caught up in some other part of the testing hence taken a pause on FTS portion of the volume test. Will be back to it soon...

            ritesh.agarwal Ritesh Agarwal added a comment - Hello Abhi Dangeti , I am caught up in some other part of the testing hence taken a pause on FTS portion of the volume test. Will be back to it soon...
            abhinav Abhi Dangeti added a comment -

            Marking this as resolved for now, as there's no other planned fixes at this time. 

            abhinav Abhi Dangeti added a comment - Marking this as resolved for now, as there's no other planned fixes at this time. 

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty