Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38303

FTS - Scorch file merger leaks index snapshot ref counts on index closure

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      During the file merge introduction sequence, the introducer bumps the new index snapshot reference count to ensure it to live until the merger acknowledges the introductions and handles( decrements) the reference count for the newly introduced index snapshot.

      ref- https://github.com/blevesearch/bleve/blob/2f21902d034941f7d00332cbd70ead50da272448/index/scorch/introducer.go#L424

      Though the merger routine sends all the file merge introductions one at a time after each merge task completion, it handles all these merge introduction acknowledgements in a single loop with a "Bail Out" check on the index Close channel between every merge acknowledgement handling where the index snapshot reference count decrements happening.

      ref- https://github.com/blevesearch/bleve/blob/05d86ea8f6e30456949f612cf68cf4a27ce8c9c5/index/scorch/merge.go#L258-L261

      Now, this creates a potential reference count leak issue with an index closure on events like cbft rollbacks. 

      Let's say, 

      1. Merger totally had 6 merge tasks and it gave merge introduction requests for all of them to the introducer.
      2. The introducer did the introductions for all of them and bumped the index snapshot reference counts for all of them.
      3. Now the merger comes out of it merge loop and started processing the acknowledgements here - https://github.com/blevesearch/bleve/blob/05d86ea8f6e30456949f612cf68cf4a27ce8c9c5/index/scorch/merge.go#L253
      4. But by this time, an index closure happened due to the rollback. And without decrementing the reference counts of all these newly introduced index snapshots the merger exits here - https://github.com/blevesearch/bleve/blob/05d86ea8f6e30456949f612cf68cf4a27ce8c9c5/index/scorch/merge.go#L257 to create a reference count leak for all the merge introduced snapshots. 
      5. The files belonging to these index snapshots, will enter into the FD `DEL` orphaned state due to the forced index directory clean ups triggered from here - https://github.com/couchbase/cbft/blob/master/pindex_bleve_rollback.go#L40

      ie - files got deleted while there were live references to them by some index snapshots and those index snapshots will never get cleaned up due to the non-zero reference count resulted by the early exit loop of the merger.

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              evgeny.makarenko Evgeny Makarenko (Inactive)
              Sreekanth Sivasankaran Sreekanth Sivasankaran (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty