Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8456

[Doc'd] Update Documentation with the recommendation that THP should be disabled on Couchbase nodes.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.5.1, 3.0
    • Component/s: None
    • Security Level: Public
    • Labels:
    • Is this a Regression?:
      Yes
    • Sprint:
      TP Sprint 15

      Description

      Nodes on which THP(Transparent Huge Pages) are enabled, we are seeing high swap usage and severe timeouts in ns_server. This may get much worse under memory pressure. So ns_server had to restart whole bunch of services because of random timeouts and in the end it reached max restart intensity and therefore terminated.

      So by default THP must be disabled and this information needs to be documented.

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          kzeller kzeller added a comment -

          Hi Alk,

          This request to document is coming outside of Couchbase. It is not clear to me whether this is something we should recommend or not (disable THP)

          Let me know if this is something we recommend or not. If we do want to recommend it, has this option been tested?

          Regards,

          Karen

          Show
          kzeller kzeller added a comment - Hi Alk, This request to document is coming outside of Couchbase. It is not clear to me whether this is something we should recommend or not (disable THP) Let me know if this is something we recommend or not. If we do want to recommend it, has this option been tested? Regards, Karen
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          It's not quite outside. I have no idea if we tested this much, but we have seen at least one customer who had issues with transparent hugepages and issues stopped without.

          This matches some of our developer testing.

          So yes I want us to strongly recommend disabling transparent hugepages.

          Text can be something like that:

          RHEL6 and other newer distributions running on physical hardware are known to have transparent hugepages feature enabled. Which is in general a measurable performance boost. However under some conditions which Couchbase Server is known to trigger it may cause severe delays in page allocations. Therefore we strongly recommend everyone with that feature to disable it.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - It's not quite outside. I have no idea if we tested this much, but we have seen at least one customer who had issues with transparent hugepages and issues stopped without. This matches some of our developer testing. So yes I want us to strongly recommend disabling transparent hugepages. Text can be something like that: RHEL6 and other newer distributions running on physical hardware are known to have transparent hugepages feature enabled. Which is in general a measurable performance boost. However under some conditions which Couchbase Server is known to trigger it may cause severe delays in page allocations. Therefore we strongly recommend everyone with that feature to disable it.
          Hide
          kzeller kzeller added a comment -

          Added to RN 2.1.0:

          <rnentry type="knownissue">

          <version ver="2.1.0a"/>

          <class id="perf"/>

          <issue type="cb" ref="MB-8456"/>

          <rntext>

          <para>
          RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent
          hugepages feature enabled. In general this can provide a measurable performance boost.
          However under some conditions that Couchbase Server is known to trigger, this
          it may cause severe delays in page allocations. Therefore we strongly recommend you disable this feature with Couchbase Server.

          </para>

          </rntext>

          </rnentry>

          Show
          kzeller kzeller added a comment - Added to RN 2.1.0: <rnentry type="knownissue"> <version ver="2.1.0a"/> <class id="perf"/> <issue type="cb" ref=" MB-8456 "/> <rntext> <para> RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. In general this can provide a measurable performance boost. However under some conditions that Couchbase Server is known to trigger, this it may cause severe delays in page allocations. Therefore we strongly recommend you disable this feature with Couchbase Server. </para> </rntext> </rnentry>
          Hide
          drigby Dave Rigby added a comment -

          I think we should make this more prominent. More specifically:

          1. This is tagged as "performance" currently but it also affects cluster stability
          2. It's a pretty serious problem, and by default RHEL-based OSs are still shipping with THP enabled, so out-of-the-box experience is poor if they don't disable this.

          Show
          drigby Dave Rigby added a comment - I think we should make this more prominent. More specifically: 1. This is tagged as "performance" currently but it also affects cluster stability 2. It's a pretty serious problem, and by default RHEL-based OSs are still shipping with THP enabled, so out-of-the-box experience is poor if they don't disable this.
          Hide
          drigby Dave Rigby added a comment -

          MB to update the installer to warn about this.

          Show
          drigby Dave Rigby added a comment - MB to update the installer to warn about this.
          Hide
          ruth Ruth Harris (Inactive) added a comment -

          Does this apply to ONLY 2.1.0?

          In 2.1.0, the Known Issue can also be noted re affecting cluster stabiity.
          If the INSTALLER needs to be updated, then this Issue should be transfered to the BUILD team (not documentation).

          This Known Issue in 2.1.0 is not carried over into subsequent releases.

          --ruth

          Show
          ruth Ruth Harris (Inactive) added a comment - Does this apply to ONLY 2.1.0? In 2.1.0, the Known Issue can also be noted re affecting cluster stabiity. If the INSTALLER needs to be updated, then this Issue should be transfered to the BUILD team (not documentation). This Known Issue in 2.1.0 is not carried over into subsequent releases. --ruth
          Hide
          pvarley Patrick Varley added a comment -

          This affects all versions of couchbase

          Show
          pvarley Patrick Varley added a comment - This affects all versions of couchbase
          Hide
          ruth Ruth Harris (Inactive) added a comment -

          Anil,

          Please confirm that this KNOWN ISSUE should be added to all of the release notes. It's already in the 2.1 RN.

          Show
          ruth Ruth Harris (Inactive) added a comment - Anil, Please confirm that this KNOWN ISSUE should be added to all of the release notes. It's already in the 2.1 RN.
          Hide
          anil Anil Kumar added a comment -

          Ruth - Its added to 2.1 release notes which is fine. We should add a *Note section in out installation section of documentation to cover this point.

          Show
          anil Anil Kumar added a comment - Ruth - Its added to 2.1 release notes which is fine. We should add a *Note section in out installation section of documentation to cover this point.
          Hide
          marija Marija Jovanovic (Inactive) added a comment -

          Added a note of the type "Important" at the top of the page
          /Install/rhel-installing.dita
          wit the following warning:
          "RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. While this can provide a measurable performance boost, under some conditions that Couchbase Server is known to trigger, this option may cause severe delays in page allocations. Therefore, it is strongly recommended to disable the transparent hugepages feature when installing Couchbase Server. "

          This content that was added to the release notes 2.1.0 and is also requested to be repeated in all subsequent releases.

          This resolution applies to the release 3.0.2.

          Show
          marija Marija Jovanovic (Inactive) added a comment - Added a note of the type "Important" at the top of the page /Install/rhel-installing.dita wit the following warning: "RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. While this can provide a measurable performance boost, under some conditions that Couchbase Server is known to trigger, this option may cause severe delays in page allocations. Therefore, it is strongly recommended to disable the transparent hugepages feature when installing Couchbase Server. " This content that was added to the release notes 2.1.0 and is also requested to be repeated in all subsequent releases. This resolution applies to the release 3.0.2.

            People

            • Assignee:
              marija Marija Jovanovic (Inactive)
              Reporter:
              skumar Saran Kumar (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile

                  Gerrit Reviews

                  There are no open Gerrit changes