Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49874

FTS: bleveMaxClauseCount reset to 1024 after restart

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.0.0, 6.6.3, 7.1.0, 7.0.2
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1787

      • Create fts cluster
      • Create a bucket and index on it
      • Run a query which hits bleveMaxClauseCount greater than 1024
      • set bleveMaxClauseCount to greater than what above query needs
      • verify the value set with conciseOptions API
      • run the same query - query runs fine
      • restart fts
      • run the same query again - query fails with bleveMaxClauseCount

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-50946 [BP MB-49874 to 6.6.x] - FTS: bleveMaxClauseCount reset to 1024 after restart

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            girish.benakappa Girish Benakappa created issue -
            girish.benakappa Girish Benakappa made changes -
            Field Original Value New Value
            Assignee Keshav Murthy [ keshav ] Abhinav Dangeti [ abhinav ]
            abhinav Abhinav Dangeti made changes -
            Affects Version/s 7.0.0 [ 17233 ]
            Affects Version/s 6.6.3 [ 17420 ]
            Affects Version/s 7.0.2 [ 18012 ]
            abhinav Abhinav Dangeti made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.1.0-1836 contains cbft commit dfb3d3b with commit message:
            MB-49874: Upon process restart, bleve options to be overwritten

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1836 contains cbft commit dfb3d3b with commit message: MB-49874 : Upon process restart, bleve options to be overwritten
            Sreekanth Sivasankaran Sreekanth Sivasankaran made changes -
            Assignee Abhinav Dangeti [ abhinav ] Sreekanth Sivasankaran [ sreekanth sivasankaran ]
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment -

            The data update functionality is broken for existing indexes that are opened in the current session. (regression from last fix)

            Steps of reproduction.

            1. Create a beer sample default index and wait for indexing to complete.
            2. Shutdown cluster run.
            3. Start the cluster again.
            4. Goto bucket and edit some text and then search the new text in the index.
            5. Confirm whether you get those new updates in the search results.

            Root cause => the original fix here made the analysis workers to reset at this point - https://github.com/couchbase/cbft/blob/5cb62de5ab305604a141258c3dbb3b7840629388/cmd/cbft/init_bleve.go#L88 

            But by the time this reset happens, many of the existing scorch indexes would have already opened with the older workers from here(https://github.com/couchbase/cbft/blob/3fc4af32e1426d8ce07af03c44cb72b3f461b509/cmd/cbft/main.go#L376) which are in a closed state now due to the reset that happened above.

            So, no further indexing/mutations would be handled in those opened/old indexes since their analysis queue are closed. And they never get linked to the new refreshed/reset analysis queue.

             

            Girish Benakappa, It is surprising that this regression got escaped in the automated weekly tests.

            We need to cover such cases in our automated tests so that these are never missed in the future.

             

             

            Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment - The data update functionality is broken for existing indexes that are opened in the current session. (regression from last fix) Steps of reproduction. Create a beer sample default index and wait for indexing to complete. Shutdown cluster run. Start the cluster again. Goto bucket and edit some text and then search the new text in the index. Confirm whether you get those new updates in the search results. Root cause => the original fix here made the analysis workers to reset at this point - https://github.com/couchbase/cbft/blob/5cb62de5ab305604a141258c3dbb3b7840629388/cmd/cbft/init_bleve.go#L88   But by the time this reset happens, many of the existing scorch indexes would have already opened with the older workers from here( https://github.com/couchbase/cbft/blob/3fc4af32e1426d8ce07af03c44cb72b3f461b509/cmd/cbft/main.go#L376 ) which are in a closed state now due to the reset that happened above. So, no further indexing/mutations would be handled in those opened/old indexes since their analysis queue are closed. And they never get linked to the new refreshed/reset analysis queue.   Girish Benakappa , It is surprising that this regression got escaped in the automated weekly tests. We need to cover such cases in our automated tests so that these are never missed in the future.    
            abhinav Abhinav Dangeti made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            girish.benakappa Girish Benakappa added a comment -

            Sreekanth Sivasankaran
            We do have a case where we mutate docs and check if mutated documents are processed. But we don't have a case where we restart couchbase before update.

            It seems this is reproducible only when we restart couchbase before update but not consistently (still checking whats. missing). Will add this flow to our suite.

            girish.benakappa Girish Benakappa added a comment - Sreekanth Sivasankaran We do have a case where we mutate docs and check if mutated documents are processed. But we don't have a case where we restart couchbase before update. It seems this is reproducible only when we restart couchbase before update but not consistently (still checking whats. missing). Will add this flow to our suite.
            Sreekanth Sivasankaran Sreekanth Sivasankaran made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            abhinav Abhinav Dangeti made changes -
            Assignee Sreekanth Sivasankaran [ sreekanth sivasankaran ] Girish Benakappa [ girish.benakappa ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.1.0-2149 contains cbft commit df0d489 with commit message:
            MB-49874 - bleveMaxClauseCount reset to 1024 after restart

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2149 contains cbft commit df0d489 with commit message: MB-49874 - bleveMaxClauseCount reset to 1024 after restart
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.1.0-2149 contains cbgt commit d30a0f4 with commit message:
            MB-49874 - bleveMaxClauseCount reset to 1024 after restart

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2149 contains cbgt commit d30a0f4 with commit message: MB-49874 - bleveMaxClauseCount reset to 1024 after restart
            abhinav Abhinav Dangeti made changes -
            Link This issue blocks CBSE-11284 [ CBSE-11284 ]
            girish.benakappa Girish Benakappa added a comment -

            Verified with 7.1.0-2202

            girish.benakappa Girish Benakappa added a comment - Verified with 7.1.0-2202
            girish.benakappa Girish Benakappa made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            lynn.straus Lynn Straus made changes -
            Labels candidate-for-6.6.6
            wayne Wayne Siu made changes -
            Link This issue is cloned by MB-50946 [ MB-50946 ]
            wayne Wayne Siu made changes -
            Link This issue backports to MB-50946 [ MB-50946 ]
            wayne Wayne Siu made changes -
            Link This issue is cloned by MB-50946 [ MB-50946 ]

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty