Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6392

Indexing 11 ddocs might push system into swap pretty quickly

    Details

      Description

      1 bucket.
      6 GB bucket, 8 gb system ram,
      10M items
      40% active and resident ratio.

      It seems like that indexing 11 ddocs with 2 views each, seems to push the system into swap, pretty fast.
      Seeing this on 2.0 large cluster, with similar setup as well.
      What are the bounds for 2.0 when someone wants to use indexes and upto how many?

      See the screenshot attached for what we are seeing, in the functional tests.

      Views:-
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d1' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.key_num);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,doc.key);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d2' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.name);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.key_num]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d3' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.email);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.name]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d4' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.city);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.email]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d5' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.country);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.city]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d6' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.realm);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.country]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d7' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.coins);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.realm]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d8' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.category);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.coins]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d9' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.achievements);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.category]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d10' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.body);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.achievements]);\n}

      "} }}'
      curl -v -X PUT -H 'Content-Type: application/json' 'http://saslbucket:saslpass@localhost:8092/saslbucket/_design/d11' \
      -d '{"views": { "v1":{"map":"function(doc)

      {\nemit(doc.key,doc.city+doc.country);\n}

      "}, "v2":{"map":"function(doc,meta)

      {\nemit(meta.id,[doc.key_num,doc.body]);\n}

      "} }}'

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        karan Karan Kumar (Inactive) created issue -
        Hide
        karan Karan Kumar (Inactive) added a comment -

        The views dont emit lot of stuff, some small properties of the document.

        Show
        karan Karan Kumar (Inactive) added a comment - The views dont emit lot of stuff, some small properties of the document.
        Hide
        karan Karan Kumar (Inactive) added a comment -

        We are going to run some more experiments to understand this better.

        All these test have been run on centos 5.6
        Run on ubuntu 10.04. Different scheduler/filesystem?

        Run with nobarrier mount option.

        Show
        karan Karan Kumar (Inactive) added a comment - We are going to run some more experiments to understand this better. All these test have been run on centos 5.6 Run on ubuntu 10.04. Different scheduler/filesystem? Run with nobarrier mount option.
        Hide
        karan Karan Kumar (Inactive) added a comment -

        From Alk:
        Even more than that (assuming I'm not insane) is O(N^2) external sorting algorithm that we accidently implemented in couch_set_view_updater. Basically it keeps up to 16 1 meg files, then merges them to 1. Then continues with creating 15 1 meg files and merges everything together again. Then again, etc... It can be seen that this is quadratic behavior.

        Show
        karan Karan Kumar (Inactive) added a comment - From Alk: Even more than that (assuming I'm not insane) is O(N^2) external sorting algorithm that we accidently implemented in couch_set_view_updater. Basically it keeps up to 16 1 meg files, then merges them to 1. Then continues with creating 15 1 meg files and merges everything together again. Then again, etc... It can be seen that this is quadratic behavior.
        farshid Farshid Ghods (Inactive) made changes -
        Field Original Value New Value
        Labels 2.0-DP3-release-notes
        farshid Farshid Ghods (Inactive) made changes -
        Labels 2.0-DP3-release-notes 2.0-beta-release-notes
        karan Karan Kumar (Inactive) made changes -
        Labels 2.0-beta-release-notes 2.0-beta-release-notes system-test
        Priority Blocker [ 1 ] Major [ 3 ]
        karan Karan Kumar (Inactive) made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        karan Karan Kumar (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            karan Karan Kumar (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes