Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44241

avgScanRate stat and consolidated index status Completion and Progress computations incorrect



    • Untriaged
    • 1
    • Unknown


      request_handler.go consolideIndexStatus() consolidates status across all partitions of a given index instance (and also confounds this with consolidating source and target of an index that is in the process of moving). As it does this it intends to compute the consolidated IndexStatus fields Completion and Progress as the unweighted average across all partitions (and sources + targets of a move), but the computations are incorrect:

      s2.Completion = (s2.Completion + status.Completion) / 2
      s2.Progress = (s2.Progress + status.Progress) / 2.0

      These always weight the most recently seen partition at 50% in the running average instead of equally weighted across all partitions. The earlier a partition is seen, the more exponentially discounted its weight will be. For example, in the final average:

      • 3 partitions will be weighted 1/4, 1/4, 1/2
      • 4 partitions will be weighted 1/8, 1/8, 1/4, 1/2
      • 5 partitions will be weighted 1/16, 1/16, 1/8, 1/4, 1/2

      et cetera. Computing the correct unweighted average cannot be done incrementally – instead the sum and count must both be captured and the average computed at the end.

      2021-09-21 update: The same problem exists for the avgScanRate (avg_scan_rate) statistic:
      scan_coordinator.go handleStats()

      for _, pid := range partitions {
      	scanRate := float64(numRowsScanned-partnStats.lastNumRowsScanned.Value()) / elapsed
      	partnStats.avgScanRate.Set(int64((scanRate + float64(partnStats.avgScanRate.Value())) / 2))


        No reviews matched the request. Check your Options in the drop-down menu of this sections header.



            kevin.cherkauer Kevin Cherkauer (Inactive)
            kevin.cherkauer Kevin Cherkauer (Inactive)
            0 Vote for this issue
            2 Start watching this issue



              Gerrit Reviews

                There are no open Gerrit changes