Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47503

[7.0.2 BP MB-46932] - Recovered instances do not compact

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      During recovery, swap() of recovered instance seems to not copy storeCtx, causing cfGetter to return gcFilter with wrong gc snapshot number intervals. Any page compact of a recovered instance will cause MVCC versions to pile up.

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-46932 Recovered instances do not compact

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            jliang John Liang created issue -
            jliang John Liang made changes -
            Field Original Value New Value
            Link This issue Clones MB-46932 [ MB-46932 ]
            jliang John Liang made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s Neo [ 17615 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            wayne Wayne Siu made changes -
            Link This issue is a backport of MB-46932 [ MB-46932 ]
            wayne Wayne Siu made changes -
            Link This issue Clones MB-46932 [ MB-46932 ]
            raju Raju Suravarjjala made changes -
            Summary [7.0.1 BP MB-46932] - Recovered instances do not compact [7.0.2 BP MB-46932] - Recovered instances do not compact
            wayne Wayne Siu made changes -
            Labels plasma approved-for-7.0.2 plasma
            wayne Wayne Siu made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]
            wayne Wayne Siu made changes -
            Affects Version/s Cheshire-Cat [ 15915 ]
            jliang John Liang made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.2-6524 contains plasma commit 9167dae with commit message:
            MB-47503: Do not continue on checkpoint recovery after recovery is done

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6524 contains plasma commit 9167dae with commit message: MB-47503 : Do not continue on checkpoint recovery after recovery is done
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.2-6524 contains plasma commit 90965bd with commit message:
            MB-47503: Do not pre-allcoate plasma instance for recovery

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6524 contains plasma commit 90965bd with commit message: MB-47503 : Do not pre-allcoate plasma instance for recovery
            ritam.sharma Ritam Sharma made changes -
            Labels approved-for-7.0.2 plasma approved-for-7.0.2 plasma request-dev-verify
            wayne Wayne Siu added a comment -

            Akhil Mundroy added a comment - 19/Aug/21 9:47 PM
            Verification Steps:

            1. Create collection, add documents, create and build index.
            2. kill indexer process.
            3. Delete all documents in collection.
            4. Monitor storage stats:
              1. compacts should increase
              2. lss_data_size should decrease
              3. total_records should tend to 0

             

            Without the fix, stats in steps (4.2) and (4.3) may not come down to expected levels due to compacts not clearing out items.

            wayne Wayne Siu added a comment - Akhil Mundroy  added a comment - 19/Aug/21 9:47 PM Verification Steps: Create collection, add documents, create and build index. kill indexer process. Delete all documents in collection. Monitor storage stats: compacts  should increase lss_data_size  should decrease total_records  should tend to 0   Without the fix, stats in steps (4.2) and (4.3) may not come down to expected levels due to compacts not clearing out items.
            wayne Wayne Siu made changes -
            Assignee John Liang [ jliang ] Vikas Chaudhary [ vikas.chaudhary ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Vikas Chaudhary [ vikas.chaudhary ] Sanjit Chauhan [ JIRAUSER25325 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Labels approved-for-7.0.2 plasma request-dev-verify approved-for-7.0.2 plasma
            sanjit.chauhan Sanjit Chauhan added a comment - - edited

            Wayne Siu: What is the correct build version here?

            There is no 7.0.2-6524 build version

            instead there is 7.0.2-6523.

             

            I tested in 7.0.2-6523 and still I can see the issue. Though I deleted all the documents in collection still total_records > 0.  

            cc: Akhil Mundroy

            sanjit.chauhan Sanjit Chauhan added a comment - - edited Wayne Siu : What is the correct build version here? There is no 7.0.2-6524 build version instead there is 7.0.2-6523.   I tested in 7.0.2-6523 and still I can see the issue. Though I deleted all the documents in collection still total_records > 0.   cc:  Akhil Mundroy
            sanjit.chauhan Sanjit Chauhan added a comment -

            John Liang: Sure, Executing the below steps to validate the issue:

            1. Create collection, add documents, create and build index.
            2. kill indexer process.
            3. Delete all documents in collection.
            4. Wait for 20+ mins. Performing mutations for the period
            5. Execute CompactAll against every indexerId
            6. Monitor storage stats:
              1. compacts should increase
              2. lss_data_size should have downward trend as compared to step 1
              3. total_records should have downward trend as compared to step 1
            sanjit.chauhan Sanjit Chauhan added a comment - John Liang : Sure, Executing the below steps to validate the issue: Create collection, add documents, create and build index. kill indexer process. Delete all documents in collection. Wait for 20+ mins. Performing mutations for the period Execute CompactAll against every indexerId Monitor storage stats: compacts  should increase lss_data_size  should have downward trend as compared to step 1 total_records  should have downward trend as compared to step 1
            sanjit.chauhan Sanjit Chauhan added a comment - - edited

            Tested the issue total_records are updating after compaction. total_records updating to mutated count value.

            i.e. Created and deleted 10k documents. Later added 100 docs.

            Total_records is showing 100 only.

            ===================================================

            Compacts stats also going up after compaction

            ====================================================

            lss_data_size count going down after compaction

             

            Marking the issue resolved

            sanjit.chauhan Sanjit Chauhan added a comment - - edited Tested the issue total_records are updating after compaction. total_records updating to mutated count value. i.e. Created and deleted 10k documents. Later added 100 docs. Total_records is showing 100 only. =================================================== Compacts stats also going up after compaction ==================================================== lss_data_size count going down after compaction   Marking the issue resolved
            sanjit.chauhan Sanjit Chauhan made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            jliang John Liang added a comment -

            sounds good. Make sure you save the test log as well as cbcollect.

            jliang John Liang added a comment - sounds good. Make sure you save the test log as well as cbcollect.

            People

              sanjit.chauhan Sanjit Chauhan
              jliang John Liang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty