Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7626

Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0, 2.0.1, 2.1.0
    • Fix Version/s: 3.0
    • Component/s: installer
    • Security Level: Public
    • Labels:
    • Environment:
      OpenVZ VPS container running Ubuntu 12.04 with a CentOS 6 OS as the host on 2.6.32 kernel.
    • Triage:
      Untriaged
    • Flagged:
      Release Note

      Description

      I have done some QA to help narrow down an issue I have discovered.

      1. Setup Couchbase from 64bit Ubunut 12 package.
      2. Next select a CouchDB bucket.
      3. Next complete the setup.

      Expected: Couchbase server and bucket functioning.

      Actual: The Couchbase server appears to just keep crashing / i.e. restarting every few seconds.

      If I remove the couchDB bucket no crashes / restart occur.
      If I create a memcached bucket only the server is functioning fine and the bucket functions as expected.

      This is on a OpenVZ VPS container running Ubuntu 12.04 with a CentOS 6 OS as the host on 2.6.32 kernel.

      I went ahead and setup couchbase on another Ubuntu 12.04 dedicated server (exact replica of the above server without the virtualization) and no issues.

      What debug output information from the logs would you like to help identify this issue?

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          creator11 creator11 added a comment -

          Ok I found the fix to this issue.

          memsup as a previous bug I found was causing a segmentation fault. Upgraded to latest erlang with erlang-os-mon 2.2.10 and issue has been resolved on an OpenVZ container. I just symlinked to the new version for couchbase to use.

          Show
          creator11 creator11 added a comment - Ok I found the fix to this issue. memsup as a previous bug I found was causing a segmentation fault. Upgraded to latest erlang with erlang-os-mon 2.2.10 and issue has been resolved on an OpenVZ container. I just symlinked to the new version for couchbase to use.
          Hide
          creator11 creator11 added a comment -

          Might be handy to add this to a knowledge base for others, since I have found 2 other people that had similar issues on the web in forums.

          Show
          creator11 creator11 added a comment - Might be handy to add this to a knowledge base for others, since I have found 2 other people that had similar issues on the web in forums.
          Hide
          chiyoung Chiyoung Seo added a comment -

          Farshid,

          Please close the bug if it's not required for the QE verification.

          Show
          chiyoung Chiyoung Seo added a comment - Farshid, Please close the bug if it's not required for the QE verification.
          Hide
          mikew Mike Wiederhold added a comment -

          This might be good to add to our documentation.

          Show
          mikew Mike Wiederhold added a comment - This might be good to add to our documentation.
          Hide
          kzeller kzeller added a comment -

          Added to 2.0.1 RN:

          In OpenVZ Linux containers, the server had crashed and restarted when you
          created a Couchbase bucket. This was due to an issue in the memsup process from Erlang.
          To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Show
          kzeller kzeller added a comment - Added to 2.0.1 RN: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.
          Hide
          kzeller kzeller added a comment -

          Added to 2.0.1 RN:

          In OpenVZ Linux containers, the server had crashed and restarted when you
          created a Couchbase bucket. This was due to an issue in the memsup process from Erlang.
          To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Show
          kzeller kzeller added a comment - Added to 2.0.1 RN: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.
          Hide
          kzeller kzeller added a comment -

          See RN 2.0.1 addition. If you reopen, please assign to engineer responsible for fixing.

          Show
          kzeller kzeller added a comment - See RN 2.0.1 addition. If you reopen, please assign to engineer responsible for fixing.
          Hide
          thuan Thuan Nguyen added a comment -

          Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/)
          MB-7626: tune rebalance options during cluster setup (Revision 7c828a6a1de4f7c8fb76265766d3eb53f183fe3e)

          Result = SUCCESS
          pavelpaulau :
          Files :

          • pytests/performance/perf.py
          Show
          thuan Thuan Nguyen added a comment - Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/ ) MB-7626 : tune rebalance options during cluster setup (Revision 7c828a6a1de4f7c8fb76265766d3eb53f183fe3e) Result = SUCCESS pavelpaulau : Files : pytests/performance/perf.py
          Hide
          perry Perry Krug added a comment -

          Karen, can you provide instructions on which version of Erlang to use here and instructions on how to apply the workaround mentioned?

          Show
          perry Perry Krug added a comment - Karen, can you provide instructions on which version of Erlang to use here and instructions on how to apply the workaround mentioned?
          Hide
          kzeller kzeller added a comment -

          Hi,

          Can you provide the version of Erlang I Ishould document, and is the workaround just to upgrade to this version? Or are there other steps someone must do in order to upgrade to Erlang using CB?

          Thanks,

          Karen

          Show
          kzeller kzeller added a comment - Hi, Can you provide the version of Erlang I Ishould document, and is the workaround just to upgrade to this version? Or are there other steps someone must do in order to upgrade to Erlang using CB? Thanks, Karen
          Hide
          Vindobona Joseph H. added a comment - - edited

          I've actually applied the the Workaround for MB-4376 successfully (at least for a development environment) to CB 2.0.1 as I didn't succeed in getting CB to use a entirely different version of Erlang.

          Show
          Vindobona Joseph H. added a comment - - edited I've actually applied the the Workaround for MB-4376 successfully (at least for a development environment) to CB 2.0.1 as I didn't succeed in getting CB to use a entirely different version of Erlang.
          Hide
          perry Perry Krug added a comment -

          Note that we have confirmed through another customer that the workaround in mb-4376 is successful.

          Can this get rolled into 2.0.2?

          Show
          perry Perry Krug added a comment - Note that we have confirmed through another customer that the workaround in mb-4376 is successful. Can this get rolled into 2.0.2?
          Hide
          kzeller kzeller added a comment -

          Hi Perry, confirm this, ignore formatting (will go in release notes, where workarounds, known issues, etc. go):

          =========

          last email from prospect:
          I’ve solved the problem by compiling memsup with the patch mentioned in http://www.couchbase.com/issues/browse/MB-4376 applied. This patch is included in the R15B01 release of Erlang. Perhaps Issue 4376 shouldn’t be marked fixed until Couchbase is using Erlang R15B01 or later?

          ==========

          Existing content:

          In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Proposed addition:

          1) You should stop Couchbase server
          2) Make a copy of your original memsup then apply the patch available on GitHub: https://github.com/vorobev/otp/compare/maint-memsup.patch

          You can also download the following two files and compile yourself:

          -https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.c

          -https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.h

          gcc memsup.c -o memsup

          Place the compiled code in /opt/membase/lib/erlang/lib/os_mon-2.2.5/priv/bin/

          3) Restart Couchbase Server

          Show
          kzeller kzeller added a comment - Hi Perry, confirm this, ignore formatting (will go in release notes, where workarounds, known issues, etc. go): ========= last email from prospect: I’ve solved the problem by compiling memsup with the patch mentioned in http://www.couchbase.com/issues/browse/MB-4376 applied. This patch is included in the R15B01 release of Erlang. Perhaps Issue 4376 shouldn’t be marked fixed until Couchbase is using Erlang R15B01 or later? ========== Existing content: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version. Proposed addition: 1) You should stop Couchbase server 2) Make a copy of your original memsup then apply the patch available on GitHub: https://github.com/vorobev/otp/compare/maint-memsup.patch You can also download the following two files and compile yourself: - https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.c - https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.h gcc memsup.c -o memsup Place the compiled code in /opt/membase/lib/erlang/lib/os_mon-2.2.5/priv/bin/ 3) Restart Couchbase Server
          Hide
          perry Perry Krug added a comment -

          That's fine from a workaround perspective I guess...but my request was more about actually fixing the bug for 2.0.2.

          I'll add the ns_server component.

          Thanks

          Show
          perry Perry Krug added a comment - That's fine from a workaround perspective I guess...but my request was more about actually fixing the bug for 2.0.2. I'll add the ns_server component. Thanks
          Show
          kzeller kzeller added a comment - Added content: https://github.com/couchbase/docs/commit/b8094b5c61802fb4b76f0ae6c9f4979c5f1aa6f4
          Show
          kzeller kzeller added a comment - Added content: https://github.com/couchbase/docs/commit/b8094b5c61802fb4b76f0ae6c9f4979c5f1aa6f4
          Hide
          perry Perry Krug added a comment -

          Karen, this isn't just about documentation, this is a bug that needs to be fixed in the product.

          Show
          perry Perry Krug added a comment - Karen, this isn't just about documentation, this is a bug that needs to be fixed in the product.
          Hide
          maria Maria McDuff (Inactive) added a comment -

          deferring out of 2.0.2

          Show
          maria Maria McDuff (Inactive) added a comment - deferring out of 2.0.2
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          We're simply waiting until we upgrade to R16-something. When that happens this issue will automatically be fixed.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - We're simply waiting until we upgrade to R16-something. When that happens this issue will automatically be fixed.
          Hide
          wayne Wayne Siu added a comment -

          Removing the release notes description in the summary, and assigning it to PM to prioritize

          Show
          wayne Wayne Siu added a comment - Removing the release notes description in the summary, and assigning it to PM to prioritize
          Hide
          schatt Drew Schatt added a comment -

          This is sort of a big issue for us - we were considering deploying couchbase on some rather beefy virtual servers here. In our case, we're using proxmox, and couchbase 2.2.0_community and 2.5.1 both have the issue... which makes them unusable.

          Show
          schatt Drew Schatt added a comment - This is sort of a big issue for us - we were considering deploying couchbase on some rather beefy virtual servers here. In our case, we're using proxmox, and couchbase 2.2.0_community and 2.5.1 both have the issue... which makes them unusable.
          Hide
          anil Anil Kumar added a comment -

          As per Alk with R16 this should be fixed.

          Show
          anil Anil Kumar added a comment - As per Alk with R16 this should be fixed.

            People

            • Assignee:
              anil Anil Kumar
              Reporter:
              creator11 creator11
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes