Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-9822

One of nodes is too slow during indexing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 3.0
    • 2.2.0, 2.5.0, 3.0
    • view-engine, XDCR
    • Security Level: Public
    • Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630
      Memory = 64 GB
      Disk = 2 x SSD

    Description

      The test case is pretty simple:
      1. Disable auto-compaction.
      2. Create initial dataset, wait for persistence and TAP replication.
      3. Compact bucket.
      4. Define design documents.
      5. Trigger index build, report total indexing time.

      There is an issue with the last step. Usually it takes 25 minutes to finish initial indexing. However occasionally it takes longer, about 75 minutes with quite misleading progress reporting.

      From logs I can see that 3 nodes finished indexing after 25 minutes:

      [couchdb:info,2014-01-05T17:16:59.926,ns_1@172.23.96.15:<0.30369.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/B`, updater finished
      [couchdb:info,2014-01-05T17:18:01.914,ns_1@172.23.96.15:<0.30380.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/D`, updater finished
      [couchdb:info,2014-01-05T17:18:25.264,ns_1@172.23.96.15:<0.30348.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/A`, updater finished
      [couchdb:info,2014-01-05T17:18:33.827,ns_1@172.23.96.15:<0.30358.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/C`, updater finished

      [couchdb:info,2014-01-05T17:18:27.555,ns_1@172.23.96.17:<0.26484.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/B`, updater finished
      [couchdb:info,2014-01-05T17:19:23.092,ns_1@172.23.96.17:<0.26494.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/D`, updater finished
      [couchdb:info,2014-01-05T17:20:08.889,ns_1@172.23.96.17:<0.26474.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/C`, updater finished
      [couchdb:info,2014-01-05T17:20:37.673,ns_1@172.23.96.17:<0.26464.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/A`, updater finished

      [couchdb:info,2014-01-05T17:18:30.404,ns_1@172.23.96.18:<0.26487.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/B`, updater finished
      [couchdb:info,2014-01-05T17:20:28.915,ns_1@172.23.96.18:<0.26497.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/D`, updater finished
      [couchdb:info,2014-01-05T17:21:02.386,ns_1@172.23.96.18:<0.26477.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/C`, updater finished
      [couchdb:info,2014-01-05T17:21:32.953,ns_1@172.23.96.18:<0.26467.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/A`, updater finished

      But one of them finished indexing only after 75 minutes:

      [couchdb:info,2014-01-05T18:06:44.109,ns_1@172.23.96.16:<0.26488.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/C`, updater finished
      [couchdb:info,2014-01-05T18:06:48.814,ns_1@172.23.96.16:<0.26508.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/D`, updater finished
      [couchdb:info,2014-01-05T18:06:53.955,ns_1@172.23.96.16:<0.26474.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/A`, updater finished
      [couchdb:info,2014-01-05T18:06:55.265,ns_1@172.23.96.16:<0.26498.0>:couch_log:info:39]Set view `bucket-1`, main group `_design/B`, updater finished

      It apparently affects total indexing time and also makes aggregated progress a little bit confusing.

      Attachments

        1. beam.smp_cpu_xdcr.jpeg
          beam.smp_cpu_xdcr.jpeg
          670 kB
        2. beam.smp_cpu.jpeg
          beam.smp_cpu.jpeg
          370 kB
        3. beam.smp_cpu.png
          beam.smp_cpu.png
          93 kB
        4. cpu_r16_b737.png
          cpu_r16_b737.png
          260 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty