Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43967

Implement Indexing/GSI getIndexStatus Optimisations for large number of indexes

    XMLWordPrintable

Details

    • 1

    Description

      At indexer side, last_scan_time stat can be excluded while computing the new ETag. Every 5-10mins, indexer can generate a new ETag, which forces ns-server to refresh the cached index status data. This will update the last scan time and also make sure things remain in sync even if there are any discrepancies while maintaining the cache at both ends.

      Attachments

        1. cksum.go
          6 kB
        2. cksum.out
          2 kB
        3. create.sh
          0.7 kB

        Issue Links

          For Gerrit Dashboard: MB-43967
          # Subject Branch Project Status CR V

          Activity

            Before-after performance tests that took advantage of Parts 1-2 (but not Part 3 external ETag) showed getIndexStatus on 1,000 indexes over three nodes sped up from ~20 per second to ~45 per second, and percentage of time spent in JSON marshal/unmarshall decreased by about 1/3. Part 3 would improve on these results even more as it avoids almost all marshaling of index stats, but right now there are more urgent tasks than testing that.

            kevin.cherkauer Kevin Cherkauer added a comment - Before-after performance tests that took advantage of Parts 1-2 (but not Part 3 external ETag) showed getIndexStatus on 1,000 indexes over three nodes sped up from ~20 per second to ~45 per second, and percentage of time spent in JSON marshal/unmarshall decreased by about 1/3. Part 3 would improve on these results even more as it avoids almost all marshaling of index stats, but right now there are more urgent tasks than testing that.
            mihir.kamdar Mihir Kamdar (Inactive) added a comment - - edited

            Kevin Cherkauer is there any functional testing required for this improvement ?

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - - edited Kevin Cherkauer is there any functional testing required for this improvement ?

            Mihir Kamdar The changes only affect internal (unpublished) REST APIs – external functionality is unchanged. I'm not sure if QE tests internal APIs. If you do, there is info how to do this with curl here:

            https://issues.couchbase.com/browse/MB-41067?focusedCommentId=481426&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-481426

            kevin.cherkauer Kevin Cherkauer added a comment - Mihir Kamdar The changes only affect internal (unpublished) REST APIs – external functionality is unchanged. I'm not sure if QE tests internal APIs. If you do, there is info how to do this with curl here: https://issues.couchbase.com/browse/MB-41067?focusedCommentId=481426&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-481426

            MB-45400 was a follow-up fix for this feature.

            kevin.cherkauer Kevin Cherkauer added a comment - MB-45400 was a follow-up fix for this feature.

            closing based on verification of https://issues.couchbase.com/browse/MB-41067

            ajay.bhullar Ajay Bhullar added a comment - closing based on verification of https://issues.couchbase.com/browse/MB-41067

            People

              ajay.bhullar Ajay Bhullar
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty