Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7737

[Doc'd] Swap usage shoots up with view querying or indexing

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.1.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Seen on the latest few builds
      Observed on:
      2.0.1-153 Linux
      4 Core 30GB SSD machines

      Description

      • Setup a 7 node cluster,
      • RAM (for each physical machine): ~32GB
      • create 2 buckets: default (bucket quota: 12000MB per node) and saslbucket (bucket quota: 7000MB per node)
      • run a load (with just creates), until the item count hits a mark
      • create 2 views over each of the buckets
      • Run load that pushes system into DGM, ~80% resident ratio for default, and ~60% resident ratio for saslbucket
      • Run queries continuously against the views as the load continues.
        (for e.g.:
        curl -X GET 'http://10.1.3.235:8092/default/_design/d1/_view/v1?startkey=0&stale=ok&limit=1000')
      • Noticed the increased swap usage.
      • Start a new mixed load on both the buckets (creates, updates, deletes, expires).
      • Setup XDCR to another 5 node cluster.
      • Continue querying, noticed considerable increase in swap usage.

      This behavior is seen with a cluster with views as well (no XDCR involved).

      1. vperf-lnx.loop_2.0.1-153-rel-enterprise_2.0.1-153-rel-enterprise_VESTA_Feb-11-2013_15-40-09.pdf
        2.54 MB
        Farshid Ghods
      1. Screen Shot 2013-02-18 at 2.38.32 PM.png
        35 kB
      2. Screen Shot 2013-02-13 at 4.46.23 PM.png
        124 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        abhinav Abhinav Dangeti added a comment -

        Setting swappiness (to zero) in linux:
        sudo sysctl vm.swappiness=0

        To see what the swappiness of the system is: either sysctl vm.swappiness or cat /proc/sys/vm/swappiness.

        I think it would be wise to note how the performance is affected with this setting.
        I'll try and set up a heavy dgm - rebalance run with and without this setting at zero, to note the effect on performance.

        Show
        abhinav Abhinav Dangeti added a comment - Setting swappiness (to zero) in linux: sudo sysctl vm.swappiness=0 To see what the swappiness of the system is: either sysctl vm.swappiness or cat /proc/sys/vm/swappiness. I think it would be wise to note how the performance is affected with this setting. I'll try and set up a heavy dgm - rebalance run with and without this setting at zero, to note the effect on performance.
        Hide
        abhinav Abhinav Dangeti added a comment -

        No significant performance regression in rebalance times with swappiness at 0 and swappiness at 10. (on build 2.1.0-715)
        swappiness=10 :: http://qa.hq.northscale.net/job/litmuses-graph-loop/246/ ..rebalance time: 437s
        swappiness=0 :: http://qa.hq.northscale.net/job/litmuses-graph-loop/245/ ..rebalance time: 461s

        So, I guess the recommendation would be to have swappiness set at zero, whenever there be a views scenario with/without involving topology changes.

        Show
        abhinav Abhinav Dangeti added a comment - No significant performance regression in rebalance times with swappiness at 0 and swappiness at 10. (on build 2.1.0-715) swappiness=10 :: http://qa.hq.northscale.net/job/litmuses-graph-loop/246/ ..rebalance time: 437s swappiness=0 :: http://qa.hq.northscale.net/job/litmuses-graph-loop/245/ ..rebalance time: 461s So, I guess the recommendation would be to have swappiness set at zero, whenever there be a views scenario with/without involving topology changes.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        hm. But sysctl or echo to /proc/sys/vm/swappiness will not hold over reboot.

        On both rhel5 and my Debian box I'm seeing /etc/sysctl.conf being used at boot time to set all sorts of sysctl variables.

        We should recommend users editing this file.

        I.e. just appending:

        vm.swappiness = 0

        (and reboot) should be enough on both all kinds of ubuntus and all kinds of rhels.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - hm. But sysctl or echo to /proc/sys/vm/swappiness will not hold over reboot. On both rhel5 and my Debian box I'm seeing /etc/sysctl.conf being used at boot time to set all sorts of sysctl variables. We should recommend users editing this file. I.e. just appending: vm.swappiness = 0 (and reboot) should be enough on both all kinds of ubuntus and all kinds of rhels.
        Hide
        kzeller kzeller added a comment -

        <rnentry>

        <version ver="2.1.0a"/>

        <version ver="2.0.0l"/>

        <version ver="2.0.0k"/>

        <class id="cluster"/>

        <issue type="cb" ref="MB-7737"/>
        <issue type="cb" ref="MB-7774"/>

        <rntext>

        <para>
        By default most Linux systems have swappiness set to 60. This will lead to overuse of disk swap with
        Couchbase Server. Please follow our current recommendations on swappiness and swap space, see
        <xref linkend="couchbase-bestpractice-cloud-swap"/>.
        </para>

        </rntext>

        </rnentry>

        Show
        kzeller kzeller added a comment - <rnentry> <version ver="2.1.0a"/> <version ver="2.0.0l"/> <version ver="2.0.0k"/> <class id="cluster"/> <issue type="cb" ref=" MB-7737 "/> <issue type="cb" ref=" MB-7774 "/> <rntext> <para> By default most Linux systems have swappiness set to 60. This will lead to overuse of disk swap with Couchbase Server. Please follow our current recommendations on swappiness and swap space, see <xref linkend="couchbase-bestpractice-cloud-swap"/>. </para> </rntext> </rnentry>
        Hide
        kzeller kzeller added a comment -

        <rnentry>

        <version ver="2.1.0a"/>

        <version ver="2.0.0l"/>

        <version ver="2.0.0k"/>

        <class id="cluster"/>

        <issue type="cb" ref="MB-7737"/>
        <issue type="cb" ref="MB-7774"/>

        <rntext>

        <para>
        By default most Linux systems have swappiness set to 60. This will lead to overuse of disk swap with
        Couchbase Server. Please follow our current recommendations on swappiness and swap space, see
        <xref linkend="couchbase-bestpractice-cloud-swap"/>.
        </para>

        </rntext>

        </rnentry>

        Show
        kzeller kzeller added a comment - <rnentry> <version ver="2.1.0a"/> <version ver="2.0.0l"/> <version ver="2.0.0k"/> <class id="cluster"/> <issue type="cb" ref=" MB-7737 "/> <issue type="cb" ref=" MB-7774 "/> <rntext> <para> By default most Linux systems have swappiness set to 60. This will lead to overuse of disk swap with Couchbase Server. Please follow our current recommendations on swappiness and swap space, see <xref linkend="couchbase-bestpractice-cloud-swap"/>. </para> </rntext> </rnentry>

          People

          • Assignee:
            kzeller kzeller
            Reporter:
            abhinav Abhinav Dangeti
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes