Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6629

Slow incremental indexing and view compaction on idle cluster

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0
    • Component/s: view-engine
    • Security Level: Public
    • Labels:
    • Environment:
      1695

      Description

      Workload:-
      Phase1
      Load 15M items, 1K value sized , wait for indexing and view compaction to complete,
      It took about ~73 minutes to create the first index.
      Phase2
      Mutate 10M items, and change value size to 1.8K.
      Wait for indexing and compaction to complete.
      There is no front end load, the cluster is just doing view compaction.
      The view compaction post initial building process is still going on. ~ 48hrs plus

      1 ddoc with 4 views in each.
      15M items, replica=1, replica_index=enabled.

      1 ddocs, 4 views in each.
      10.6.2.7:8091

      Indexes and database files live separately, with dedicated disks.

      Some disk benchmarks on index partition:-
      https://friendpaste.com/1JxnXjI6YcsQtFLSAWqYa6
      https://friendpaste.com/1JxnXjI6YcsQtFLSAWqXsO

      node 7
      8.5G /data2/@indexes/default/main_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.2
      7.7G /data2/@indexes/default/replica_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.1
      6.1G /data2/@indexes/default/replica_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.1.compact
      257M /data2/@indexes/default/replica_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.log
      20K /data2/@indexes/default/tmp_02dcc6636d5e6f2bb68fd0fc592ba9a0_main
      16K /data2/@indexes/default/tmp_02dcc6636d5e6f2bb68fd0fc592ba9a0_replica

      node 11
      11G /data2/@indexes/default/main_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.1
      8.9G /data2/@indexes/default/main_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.1.compact
      565M /data2/@indexes/default/main_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.log
      8.8G /data2/@indexes/default/replica_02dcc6636d5e6f2bb68fd0fc592ba9a0.view.1
      20K /data2/@indexes/default/tmp_02dcc6636d5e6f2bb68fd0fc592ba9a0_main
      20K /data2/@indexes/default/tmp_02dcc6636d5e6f2bb68fd0fc592ba9a0_replica

      1. pidstat_node7.log
        61 kB
        Karan Kumar
      2. pidstats_node11.log
        65 kB
        Karan Kumar
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        karan Karan Kumar (Inactive) added a comment -

        You can look at the atop values from Sept 13.

        Show
        karan Karan Kumar (Inactive) added a comment - You can look at the atop values from Sept 13.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Wasn't able to reproduce on physical machine.

        Show
        pavelpaulau Pavel Paulau added a comment - Wasn't able to reproduce on physical machine.
        Hide
        karan Karan Kumar (Inactive) added a comment -

        For now, I am going to assign this back to myself to try with latest builds on amazon ec2 environment.

        Show
        karan Karan Kumar (Inactive) added a comment - For now, I am going to assign this back to myself to try with latest builds on amazon ec2 environment.
        Show
        farshid Farshid Ghods (Inactive) added a comment - related to http://www.couchbase.com/issues/browse/CBD-475
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #515 (See http://qa.hq.northscale.net/job/github-couchdb-preview/515/)
        Revert "MB-6629 Faster index compaction retries" (Revision 4a1882b75b364d959b82100fe7971afc98fc322b)

        Result = SUCCESS
        Farshid Ghods :
        Files :

        • src/couch_set_view/src/couch_set_view_updater.erl
        • src/couch_set_view/src/couch_set_view_group.erl
        • src/couch_set_view/src/couch_set_view_compactor.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #515 (See http://qa.hq.northscale.net/job/github-couchdb-preview/515/ ) Revert " MB-6629 Faster index compaction retries" (Revision 4a1882b75b364d959b82100fe7971afc98fc322b) Result = SUCCESS Farshid Ghods : Files : src/couch_set_view/src/couch_set_view_updater.erl src/couch_set_view/src/couch_set_view_group.erl src/couch_set_view/src/couch_set_view_compactor.erl

          People

          • Assignee:
            karan Karan Kumar (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes