Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62015

Recovery index time is increased by 18 times (1800%)

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown
    • Plasma-June-3-2024, Plasma-June-10-2024

    Description

      Hi team, 
      Seems like with recent changes in plasma in 7.6.2 and its BP of 7.2.6 the recovery index time is regressed by 1800%. (0.3 min --> 5.8 mins) for 250M docs.


      +
      7.6.2 release:+
      https://perf.jenkins.couchbase.com/job/secondary-plasma-dgm/7604/ 7.6.2-3623
      https://perf.jenkins.couchbase.com/job/secondary-plasma-dgm/7628/ 7.6.2-3649

      7.2.6 release: 
      https://perf.jenkins.couchbase.com/job/secondary-plasma-dgm/7612/ 7.2.6-8025
      https://perf.jenkins.couchbase.com/job/secondary-plasma-dgm/7635/ 7.2.6-8033

      The only changes went between 7.2.6-8025 and 7.2.6-8033 are related to plasma in recovery index cleanup which are back port of original commit that went in 7.6.2-3623 and 7.6.2-3649 in 7.6.2-3636 to be exact. 

      7.6.2
      https://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.6.2&fromBuild=3623&toVersion=7.6.2&toBuild=3649&f_asterixdb=off&f_backup=off&f_cbas=off&f_cbas-core=off&f_cbbs=off&f_cbft=off&f_cbftx=off&f_cbgt=off&f_couchbase-cli=off&f_eventing=off&f_goxdcr=off&f_indexing=off&f_kv_engine=off&f_n1fty=off&f_ns_server=off&f_plasma=on&f_product-metadata=off&f_query=off&f_query-ee=off&f_query-ui=off&f_regulator=off&f_testrunner=off 

      7.2.6
      https://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.2.6&fromBuild=8025&toVersion=7.2.6&toBuild=8033&f_asterixdb=off&f_cbas=off&f_cbas-core=off&f_kv_engine=off&f_ns_server=off&f_plasma=on&f_query=off 


       
      CHANGELOG for plasma:
      Commit: 33fde2f764ea1e4d8720a54261489a9995fc6c4c in build: couchbase-server-7.2.6-8030
      MB-61867 [BP 7.2.6]: Validate offset before relocating in recovery cleaner Change-Id: Ie9a68d53ea60107b6a29d670ee4acc448005d683
      Author: Durga Akhil Mundroy <akhil.m.durga2@gmail.com>
      Committer: Durga Akhil Mundroy <akhil.m.durga2@gmail.com>


      Commit: 4b45bcd28065957cc30acbedeceed6e0e5916efd in build: couchbase-server-7.2.6-8030
      MB-61867 [BP 7.2.6]: Validate offset before relocating in log cleaner Change-Id: Ifa2d596a9104301fbc7d7409c9177a781363ce8e
      Author: Durga Akhil Mundroy <akhil.m.durga2@gmail.com>
      Committer: Durga Akhil Mundroy <akhil.m.durga2@gmail.com>


      Logs for 7.2.6: 
      7.2.6-8025: good one
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.10.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.3.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.4.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.5.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.6.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7612/172.23.97.7.zip

      7.2.6-8033 bad one : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.10.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.3.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.4.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.5.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.6.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-dgm-7635/172.23.97.7.zip 

      Attachments

        Issue Links

          Activity

            People

              akhil.mundroy Akhil Mundroy
              devansh.srivastava Devansh Srivastava
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty