Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43796

OSO Enabled Initial Index build time greater than OSO Disabled

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.1.0
    • secondary-index
    • 7.0.0-4253
    • Untriaged
    • 1
    • Unknown

    Description

      On running a 1000 Index test on Plasma with OSO Enabled , we see that the time for initial build of indexes is more than that with OSO Disabled 

      The test config is as follows : 

      1. Create a bucket with 50M Docs on 100 Collections at ~20% Resident Ratio (KV)
      2. Create 1000 Indexes using PLASMA 
      3. Calculate time to build these 1000 Indexes

       

      Results are as follows : 

       

      Bucket RR OSO Initial Build Time Job
      ~20% Enabled 20.3mins http://perf.jenkins.couchbase.com/job/hemera/744/ 
      ~20% Disabled 10.8mins http://perf.jenkins.couchbase.com/job/hemera/745/ 
      100% Enabled 19.9mins http://perf.jenkins.couchbase.com/job/hemera/747/ 
      100% Disabled 10.6mins http://perf.jenkins.couchbase.com/job/hemera/746/ 

      All runs are on 7.0.0-4253.

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            1. Indexer needs to process a lot more mutations in OSO mode vs non-OSO. 

            Below is the projector ENDP stat for MAINT_STREAM after the test finishes.

             OSO

            2021-11-10T12:24:54.540-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,d4c4)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_a911101fa0309c36044eb6532996eaa9] stats: {"mutCount":9908014,"upsertCount":40317600,"deleteCount":0,"upsdelCount":0,"syncCount":81,"beginCount":205,"endCount":0,"snapCount":205,"flushCount":126901,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":5743048,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":17770,"latency.max":6711969895,"latency.avg":711535477,"latency.movingAvg":136782628,"endpChLen":0}
            

            Non-OSO

             2021-11-10T13:06:27.061-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,9e46)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_e81c7d5409fd5b5e20e168db1312a092] stats: {"mutCount":280,"upsertCount":0,"deleteCount":0,"upsdelCount":0,"syncCount":75,"beginCount":205,"endCount":0,"snapCount":0,"flushCount":275,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":0,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":9223372036854775807,"latency.max":-9223372036854775808,"latency.avg":0,"latency.movingAvg":0,"endpChLen":0}
            

            2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again.

            OSO

            2021-11-10T11:57:45.342-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, s
            copeId: , collectionId: , vbuckets: 1024 Crc64: 12148381171197068428 snapType DISK_SNAP -
                {vbno, vbuuid, manifest, seqno, oscount, snapshot-start, snapshot-end}
                {    0     9a11417a8366 1        596        490        596        596}
                {    1     b612eda54df6 1        586        480        586        586}
                {    2     b1a1b2159ff9 1        593        479        593        593}
                {    3     183d014cfb3f 1        626        498        626        626}
                {    4     93dddf3e6c06 1        593        487        593        593}
                {    5     ae2da4b54760 1        620        494        620        620}
                {    6     e8d31d437176 1        622        501        622        622}
                {    7     240eb47a523b 1        615        492        615        615}
                {    8     4dec1f91ca80 1        616        491        616        616}
            

            Non-OSO

            2021-11-10T12:41:15.956-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, scopeId: , collectionId: , vbuckets: 1024 Crc64: 2435956341701773187 snapType DISK_SNAP -
                {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}
                {    0     4f7ed0f5e57f 1      49002          0      49002}
                {    1     6913c5f29bf6 1      48002          0      48002}
                {    2     774deaee7621 1      47902          0      47902}
                {    3     476e04b434ab 1      49802          0      49802}
                {    4     4874cc8df011 1      48702          0      48702}
                {    5     e3da329ebd1f 1      49402          0      49402}
                {    6     487288c0c025 1      50102          0      50102}
                {    7     254f9ff6bf89 1      49202          0      49202}
                {    8     20e3bb0ed1f4 1      49102          0      49102}
            

            deepkaran.salooja Deepkaran Salooja added a comment - 1. Indexer needs to process a lot more mutations in OSO mode vs non-OSO.  Below is the projector ENDP stat for MAINT_STREAM after the test finishes.   OSO 2021-11-10T12:24:54.540-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,d4c4)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_a911101fa0309c36044eb6532996eaa9] stats: {"mutCount":9908014,"upsertCount":40317600,"deleteCount":0,"upsdelCount":0,"syncCount":81,"beginCount":205,"endCount":0,"snapCount":205,"flushCount":126901,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":5743048,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":17770,"latency.max":6711969895,"latency.avg":711535477,"latency.movingAvg":136782628,"endpChLen":0} Non-OSO  2021-11-10T13:06:27.061-08:00 [Info] ENDP[<-(cen-s705.perf.couchbase.com:9105,9e46)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_e81c7d5409fd5b5e20e168db1312a092] stats: {"mutCount":280,"upsertCount":0,"deleteCount":0,"upsdelCount":0,"syncCount":75,"beginCount":205,"endCount":0,"snapCount":0,"flushCount":275,"collectionCreate":0,"collectionDrop":0,"collectionFlush":0,"scopeCreate":0,"scopeDrop":0,"collectionChanged":0,"updateSeqno":0,"seqnoAdvanced":0,"osoSnapshotStart":0,"osoSnapshotEnd":0,"latency.min":9223372036854775807,"latency.max":-9223372036854775808,"latency.avg":0,"latency.movingAvg":0,"endpChLen":0} 2. This is due to MAINT_STREAM getting started with much lower seqno in OSO mode, leading to MAINT_STREAM having to process almost all of the mutations again. OSO 2021-11-10T11:57:45.342-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, s copeId: , collectionId: , vbuckets: 1024 Crc64: 12148381171197068428 snapType DISK_SNAP - {vbno, vbuuid, manifest, seqno, oscount, snapshot-start, snapshot-end} { 0 9a11417a8366 1 596 490 596 596} { 1 b612eda54df6 1 586 480 586 586} { 2 b1a1b2159ff9 1 593 479 593 593} { 3 183d014cfb3f 1 626 498 626 626} { 4 93dddf3e6c06 1 593 487 593 593} { 5 ae2da4b54760 1 620 494 620 620} { 6 e8d31d437176 1 622 501 622 622} { 7 240eb47a523b 1 615 492 615 615} { 8 4dec1f91ca80 1 616 491 616 616} Non-OSO 2021-11-10T12:41:15.956-08:00 [Info] Indexer::startKeyspaceIdStream Stream: MAINT_STREAM KeyspaceId: bucket-1 SessionId 1 RestartTS bucket: bucket-1, scopeId: , collectionId: , vbuckets: 1024 Crc64: 2435956341701773187 snapType DISK_SNAP - {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end} { 0 4f7ed0f5e57f 1 49002 0 49002} { 1 6913c5f29bf6 1 48002 0 48002} { 2 774deaee7621 1 47902 0 47902} { 3 476e04b434ab 1 49802 0 49802} { 4 4874cc8df011 1 48702 0 48702} { 5 e3da329ebd1f 1 49402 0 49402} { 6 487288c0c025 1 50102 0 50102} { 7 254f9ff6bf89 1 49202 0 49202} { 8 20e3bb0ed1f4 1 49102 0 49102}

            Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message:
            MB-43796 use true_with_seqno_advanced for OSO mode

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1836 contains indexing commit 0088b4f with commit message: MB-43796 use true_with_seqno_advanced for OSO mode

            Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message:
            MB-43796 enable OSO after cluster upgrade to 7.1

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1836 contains indexing commit 739da27 with commit message: MB-43796 enable OSO after cluster upgrade to 7.1

            Vikas Chaudhary, please retest with 7.1.0-1836 or later.

            deepkaran.salooja Deepkaran Salooja added a comment - Vikas Chaudhary , please retest with 7.1.0-1836 or later.

            Now both OSO enabled and disabled time is 12.4 

            vikas.chaudhary Vikas Chaudhary added a comment - Now both OSO enabled and disabled time is 12.4 

            People

              vikas.chaudhary Vikas Chaudhary
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty