Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8456

[Doc'd] Update Documentation with the recommendation that THP should be disabled on Couchbase nodes.

    Details

    • Is this a Regression?:
      Yes
    • Sprint:
      TP Sprint 15

      Description

      Nodes on which THP(Transparent Huge Pages) are enabled, we are seeing high swap usage and severe timeouts in ns_server. This may get much worse under memory pressure. So ns_server had to restart whole bunch of services because of random timeouts and in the end it reached max restart intensity and therefore terminated.

      So by default THP must be disabled and this information needs to be documented.

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          skumar Saran Kumar (Inactive) created issue -
          maria Maria McDuff (Inactive) made changes -
          Field Original Value New Value
          Fix Version/s 2.1.0 [ 10418 ]
          kzeller kzeller made changes -
          Summary Update Documentation with the recommendation that THP should be disabled on Couchbase nodes. [RN 2.1] Update Documentation with the recommendation that THP should be disabled on Couchbase nodes.
          kzeller kzeller made changes -
          Rank Ranked higher
          kzeller kzeller made changes -
          Labels info-request
          Hide
          kzeller kzeller added a comment -

          Hi Alk,

          This request to document is coming outside of Couchbase. It is not clear to me whether this is something we should recommend or not (disable THP)

          Let me know if this is something we recommend or not. If we do want to recommend it, has this option been tested?

          Regards,

          Karen

          Show
          kzeller kzeller added a comment - Hi Alk, This request to document is coming outside of Couchbase. It is not clear to me whether this is something we should recommend or not (disable THP) Let me know if this is something we recommend or not. If we do want to recommend it, has this option been tested? Regards, Karen
          kzeller kzeller made changes -
          Assignee Karen Zeller [ kzeller ] Aleksey Kondratenko [ alkondratenko ]
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          It's not quite outside. I have no idea if we tested this much, but we have seen at least one customer who had issues with transparent hugepages and issues stopped without.

          This matches some of our developer testing.

          So yes I want us to strongly recommend disabling transparent hugepages.

          Text can be something like that:

          RHEL6 and other newer distributions running on physical hardware are known to have transparent hugepages feature enabled. Which is in general a measurable performance boost. However under some conditions which Couchbase Server is known to trigger it may cause severe delays in page allocations. Therefore we strongly recommend everyone with that feature to disable it.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - It's not quite outside. I have no idea if we tested this much, but we have seen at least one customer who had issues with transparent hugepages and issues stopped without. This matches some of our developer testing. So yes I want us to strongly recommend disabling transparent hugepages. Text can be something like that: RHEL6 and other newer distributions running on physical hardware are known to have transparent hugepages feature enabled. Which is in general a measurable performance boost. However under some conditions which Couchbase Server is known to trigger it may cause severe delays in page allocations. Therefore we strongly recommend everyone with that feature to disable it.
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Karen Zeller [ kzeller ]
          Hide
          kzeller kzeller added a comment -

          Added to RN 2.1.0:

          <rnentry type="knownissue">

          <version ver="2.1.0a"/>

          <class id="perf"/>

          <issue type="cb" ref="MB-8456"/>

          <rntext>

          <para>
          RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent
          hugepages feature enabled. In general this can provide a measurable performance boost.
          However under some conditions that Couchbase Server is known to trigger, this
          it may cause severe delays in page allocations. Therefore we strongly recommend you disable this feature with Couchbase Server.

          </para>

          </rntext>

          </rnentry>

          Show
          kzeller kzeller added a comment - Added to RN 2.1.0: <rnentry type="knownissue"> <version ver="2.1.0a"/> <class id="perf"/> <issue type="cb" ref=" MB-8456 "/> <rntext> <para> RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. In general this can provide a measurable performance boost. However under some conditions that Couchbase Server is known to trigger, this it may cause severe delays in page allocations. Therefore we strongly recommend you disable this feature with Couchbase Server. </para> </rntext> </rnentry>
          kzeller kzeller made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          kzeller kzeller made changes -
          Summary [RN 2.1] Update Documentation with the recommendation that THP should be disabled on Couchbase nodes. [Doc'd] Update Documentation with the recommendation that THP should be disabled on Couchbase nodes.
          kzeller kzeller made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          drigby Dave Rigby added a comment -

          I think we should make this more prominent. More specifically:

          1. This is tagged as "performance" currently but it also affects cluster stability
          2. It's a pretty serious problem, and by default RHEL-based OSs are still shipping with THP enabled, so out-of-the-box experience is poor if they don't disable this.

          Show
          drigby Dave Rigby added a comment - I think we should make this more prominent. More specifically: 1. This is tagged as "performance" currently but it also affects cluster stability 2. It's a pretty serious problem, and by default RHEL-based OSs are still shipping with THP enabled, so out-of-the-box experience is poor if they don't disable this.
          drigby Dave Rigby made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Assignee kzeller [ kzeller ] Ruth Harris [ ruth ]
          drigby Dave Rigby made changes -
          Link This issue relates to MB-9193 [ MB-9193 ]
          Hide
          drigby Dave Rigby added a comment -

          MB to update the installer to warn about this.

          Show
          drigby Dave Rigby added a comment - MB to update the installer to warn about this.
          Hide
          ruth Ruth Harris added a comment -

          Does this apply to ONLY 2.1.0?

          In 2.1.0, the Known Issue can also be noted re affecting cluster stabiity.
          If the INSTALLER needs to be updated, then this Issue should be transfered to the BUILD team (not documentation).

          This Known Issue in 2.1.0 is not carried over into subsequent releases.

          --ruth

          Show
          ruth Ruth Harris added a comment - Does this apply to ONLY 2.1.0? In 2.1.0, the Known Issue can also be noted re affecting cluster stabiity. If the INSTALLER needs to be updated, then this Issue should be transfered to the BUILD team (not documentation). This Known Issue in 2.1.0 is not carried over into subsequent releases. --ruth
          Hide
          pvarley Patrick Varley added a comment -

          This affects all versions of couchbase

          Show
          pvarley Patrick Varley added a comment - This affects all versions of couchbase
          pvarley Patrick Varley made changes -
          Fix Version/s 3.0 [ 10414 ]
          Fix Version/s 2.5.1 [ 11330 ]
          Fix Version/s 2.1.1 [ 10908 ]
          Fix Version/s 2.2.0 [ 10620 ]
          Fix Version/s 2.0.1 [ 10399 ]
          Hide
          ruth Ruth Harris added a comment -

          Anil,

          Please confirm that this KNOWN ISSUE should be added to all of the release notes. It's already in the 2.1 RN.

          Show
          ruth Ruth Harris added a comment - Anil, Please confirm that this KNOWN ISSUE should be added to all of the release notes. It's already in the 2.1 RN.
          ruth Ruth Harris made changes -
          Assignee Ruth Harris [ ruth ] Anil Kumar [ anil ]
          Hide
          anil Anil Kumar added a comment -

          Ruth - Its added to 2.1 release notes which is fine. We should add a *Note section in out installation section of documentation to cover this point.

          Show
          anil Anil Kumar added a comment - Ruth - Its added to 2.1 release notes which is fine. We should add a *Note section in out installation section of documentation to cover this point.
          anil Anil Kumar made changes -
          Assignee Anil Kumar [ anil ] Ruth Harris [ ruth ]
          Is this a Regression? Yes [ 10450 ]
          ruth Ruth Harris made changes -
          Assignee Ruth Harris [ ruth ] marija jovanovic [ marija ]
          akurtzman Amy Kurtzman made changes -
          Sprint TP Sprint 15 [ 187 ]
          akurtzman Amy Kurtzman made changes -
          Rank Ranked lower
          ruth Ruth Harris made changes -
          Labels info-request install
          Hide
          marija Marija Jovanovic added a comment -

          Added a note of the type "Important" at the top of the page
          /Install/rhel-installing.dita
          wit the following warning:
          "RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. While this can provide a measurable performance boost, under some conditions that Couchbase Server is known to trigger, this option may cause severe delays in page allocations. Therefore, it is strongly recommended to disable the transparent hugepages feature when installing Couchbase Server. "

          This content that was added to the release notes 2.1.0 and is also requested to be repeated in all subsequent releases.

          This resolution applies to the release 3.0.2.

          Show
          marija Marija Jovanovic added a comment - Added a note of the type "Important" at the top of the page /Install/rhel-installing.dita wit the following warning: "RHEL6 and other newer Linux distributions running on physical hardware are known to have transparent hugepages feature enabled. While this can provide a measurable performance boost, under some conditions that Couchbase Server is known to trigger, this option may cause severe delays in page allocations. Therefore, it is strongly recommended to disable the transparent hugepages feature when installing Couchbase Server. " This content that was added to the release notes 2.1.0 and is also requested to be repeated in all subsequent releases. This resolution applies to the release 3.0.2.
          marija Marija Jovanovic made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          marija Marija Jovanovic made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              marija Marija Jovanovic
              Reporter:
              skumar Saran Kumar (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile

                  Gerrit Reviews

                  There are no open Gerrit changes