Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7626

Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0, 2.0.1, 2.1.0
    • Fix Version/s: 3.0
    • Component/s: installer
    • Security Level: Public
    • Labels:
    • Environment:
      OpenVZ VPS container running Ubuntu 12.04 with a CentOS 6 OS as the host on 2.6.32 kernel.
    • Triage:
      Untriaged
    • Flagged:
      Release Note

      Description

      I have done some QA to help narrow down an issue I have discovered.

      1. Setup Couchbase from 64bit Ubunut 12 package.
      2. Next select a CouchDB bucket.
      3. Next complete the setup.

      Expected: Couchbase server and bucket functioning.

      Actual: The Couchbase server appears to just keep crashing / i.e. restarting every few seconds.

      If I remove the couchDB bucket no crashes / restart occur.
      If I create a memcached bucket only the server is functioning fine and the bucket functions as expected.

      This is on a OpenVZ VPS container running Ubuntu 12.04 with a CentOS 6 OS as the host on 2.6.32 kernel.

      I went ahead and setup couchbase on another Ubuntu 12.04 dedicated server (exact replica of the above server without the virtualization) and no issues.

      What debug output information from the logs would you like to help identify this issue?

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          creator11 creator11 created issue -
          Hide
          creator11 creator11 added a comment -

          Ok I found the fix to this issue.

          memsup as a previous bug I found was causing a segmentation fault. Upgraded to latest erlang with erlang-os-mon 2.2.10 and issue has been resolved on an OpenVZ container. I just symlinked to the new version for couchbase to use.

          Show
          creator11 creator11 added a comment - Ok I found the fix to this issue. memsup as a previous bug I found was causing a segmentation fault. Upgraded to latest erlang with erlang-os-mon 2.2.10 and issue has been resolved on an OpenVZ container. I just symlinked to the new version for couchbase to use.
          Hide
          creator11 creator11 added a comment -

          Might be handy to add this to a knowledge base for others, since I have found 2 other people that had similar issues on the web in forums.

          Show
          creator11 creator11 added a comment - Might be handy to add this to a knowledge base for others, since I have found 2 other people that had similar issues on the web in forums.
          Hide
          chiyoung Chiyoung Seo added a comment -

          Farshid,

          Please close the bug if it's not required for the QE verification.

          Show
          chiyoung Chiyoung Seo added a comment - Farshid, Please close the bug if it's not required for the QE verification.
          chiyoung Chiyoung Seo made changes -
          Field Original Value New Value
          Assignee Chiyoung Seo [ chiyoung ] Farshid Ghods [ farshid ]
          farshid Farshid Ghods (Inactive) made changes -
          Fix Version/s .next [ 10342 ]
          mikew Mike Wiederhold made changes -
          Sprint Status Current Sprint [ 10027 ]
          Hide
          mikew Mike Wiederhold added a comment -

          This might be good to add to our documentation.

          Show
          mikew Mike Wiederhold added a comment - This might be good to add to our documentation.
          mikew Mike Wiederhold made changes -
          Assignee Farshid Ghods [ farshid ] Karen Zeller [ kzeller ]
          mikew Mike Wiederhold made changes -
          Planned End (re-schedule end date based on new assignee)
          mikew Mike Wiederhold made changes -
          Component/s documentation [ 10012 ]
          Component/s couchbase-bucket [ 10173 ]
          kzeller kzeller made changes -
          Summary Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment [RN 2.0.1] Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment
          Labels 2.0.1-release-notes
          Flagged Release Note [ 10010 ]
          Hide
          kzeller kzeller added a comment -

          Added to 2.0.1 RN:

          In OpenVZ Linux containers, the server had crashed and restarted when you
          created a Couchbase bucket. This was due to an issue in the memsup process from Erlang.
          To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Show
          kzeller kzeller added a comment - Added to 2.0.1 RN: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.
          kzeller kzeller made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kzeller kzeller added a comment -

          Added to 2.0.1 RN:

          In OpenVZ Linux containers, the server had crashed and restarted when you
          created a Couchbase bucket. This was due to an issue in the memsup process from Erlang.
          To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Show
          kzeller kzeller added a comment - Added to 2.0.1 RN: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.
          kzeller kzeller made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          mikew Mike Wiederhold made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          mikew Mike Wiederhold made changes -
          Fix Version/s 2.0.1 [ 10399 ]
          Fix Version/s .next [ 10342 ]
          mikew Mike Wiederhold made changes -
          Planned Start (set to new fixed version's start date)
          Planned End (set to new fixed version's start date)
          mikew Mike Wiederhold made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kzeller kzeller added a comment -

          See RN 2.0.1 addition. If you reopen, please assign to engineer responsible for fixing.

          Show
          kzeller kzeller added a comment - See RN 2.0.1 addition. If you reopen, please assign to engineer responsible for fixing.
          kzeller kzeller made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          thuan Thuan Nguyen added a comment -

          Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/)
          MB-7626: tune rebalance options during cluster setup (Revision 7c828a6a1de4f7c8fb76265766d3eb53f183fe3e)

          Result = SUCCESS
          pavelpaulau :
          Files :

          • pytests/performance/perf.py
          Show
          thuan Thuan Nguyen added a comment - Integrated in ui-testing #11 (See http://qa.hq.northscale.net/job/ui-testing/11/ ) MB-7626 : tune rebalance options during cluster setup (Revision 7c828a6a1de4f7c8fb76265766d3eb53f183fe3e) Result = SUCCESS pavelpaulau : Files : pytests/performance/perf.py
          Hide
          perry Perry Krug added a comment -

          Karen, can you provide instructions on which version of Erlang to use here and instructions on how to apply the workaround mentioned?

          Show
          perry Perry Krug added a comment - Karen, can you provide instructions on which version of Erlang to use here and instructions on how to apply the workaround mentioned?
          perry Perry Krug made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          kzeller kzeller added a comment -

          Hi,

          Can you provide the version of Erlang I Ishould document, and is the workaround just to upgrade to this version? Or are there other steps someone must do in order to upgrade to Erlang using CB?

          Thanks,

          Karen

          Show
          kzeller kzeller added a comment - Hi, Can you provide the version of Erlang I Ishould document, and is the workaround just to upgrade to this version? Or are there other steps someone must do in order to upgrade to Erlang using CB? Thanks, Karen
          kzeller kzeller made changes -
          Assignee Karen Zeller [ kzeller ] Chiyoung Seo [ chiyoung ]
          kzeller kzeller made changes -
          Planned End (re-schedule end date based on new assignee)
          Hide
          Vindobona Joseph H. added a comment - - edited

          I've actually applied the the Workaround for MB-4376 successfully (at least for a development environment) to CB 2.0.1 as I didn't succeed in getting CB to use a entirely different version of Erlang.

          Show
          Vindobona Joseph H. added a comment - - edited I've actually applied the the Workaround for MB-4376 successfully (at least for a development environment) to CB 2.0.1 as I didn't succeed in getting CB to use a entirely different version of Erlang.
          perry Perry Krug made changes -
          Link This issue is duplicated by MB-4376 [ MB-4376 ]
          Hide
          perry Perry Krug added a comment -

          Note that we have confirmed through another customer that the workaround in mb-4376 is successful.

          Can this get rolled into 2.0.2?

          Show
          perry Perry Krug added a comment - Note that we have confirmed through another customer that the workaround in mb-4376 is successful. Can this get rolled into 2.0.2?
          Hide
          kzeller kzeller added a comment -

          Hi Perry, confirm this, ignore formatting (will go in release notes, where workarounds, known issues, etc. go):

          =========

          last email from prospect:
          I’ve solved the problem by compiling memsup with the patch mentioned in http://www.couchbase.com/issues/browse/MB-4376 applied. This patch is included in the R15B01 release of Erlang. Perhaps Issue 4376 shouldn’t be marked fixed until Couchbase is using Erlang R15B01 or later?

          ==========

          Existing content:

          In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version.

          Proposed addition:

          1) You should stop Couchbase server
          2) Make a copy of your original memsup then apply the patch available on GitHub: https://github.com/vorobev/otp/compare/maint-memsup.patch

          You can also download the following two files and compile yourself:

          -https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.c

          -https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.h

          gcc memsup.c -o memsup

          Place the compiled code in /opt/membase/lib/erlang/lib/os_mon-2.2.5/priv/bin/

          3) Restart Couchbase Server

          Show
          kzeller kzeller added a comment - Hi Perry, confirm this, ignore formatting (will go in release notes, where workarounds, known issues, etc. go): ========= last email from prospect: I’ve solved the problem by compiling memsup with the patch mentioned in http://www.couchbase.com/issues/browse/MB-4376 applied. This patch is included in the R15B01 release of Erlang. Perhaps Issue 4376 shouldn’t be marked fixed until Couchbase is using Erlang R15B01 or later? ========== Existing content: In OpenVZ Linux containers, the server had crashed and restarted when you created a Couchbase bucket. This was due to an issue in the memsup process from Erlang. To fix this issue, you should upgrade to the latest version of Erlang, and have Couchbase Server use this version. Proposed addition: 1) You should stop Couchbase server 2) Make a copy of your original memsup then apply the patch available on GitHub: https://github.com/vorobev/otp/compare/maint-memsup.patch You can also download the following two files and compile yourself: - https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.c - https://raw.github.com/vorobev/otp/maint-memsup/lib/os_mon/c_src/memsup.h gcc memsup.c -o memsup Place the compiled code in /opt/membase/lib/erlang/lib/os_mon-2.2.5/priv/bin/ 3) Restart Couchbase Server
          Hide
          perry Perry Krug added a comment -

          That's fine from a workaround perspective I guess...but my request was more about actually fixing the bug for 2.0.2.

          I'll add the ns_server component.

          Thanks

          Show
          perry Perry Krug added a comment - That's fine from a workaround perspective I guess...but my request was more about actually fixing the bug for 2.0.2. I'll add the ns_server component. Thanks
          perry Perry Krug made changes -
          Component/s ns_server [ 10019 ]
          perry Perry Krug made changes -
          Assignee Chiyoung Seo [ chiyoung ] Aleksey Kondratenko [ alkondratenko ]
          perry Perry Krug made changes -
          Planned End (re-schedule end date based on new assignee)
          perry Perry Krug made changes -
          Affects Version/s 2.0.1 [ 10399 ]
          kzeller kzeller made changes -
          Rank Ranked higher
          kzeller kzeller made changes -
          Assignee Aleksey Kondratenko [ alkondratenko ] Karen Zeller [ kzeller ]
          Show
          kzeller kzeller added a comment - Added content: https://github.com/couchbase/docs/commit/b8094b5c61802fb4b76f0ae6c9f4979c5f1aa6f4
          kzeller kzeller made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Show
          kzeller kzeller added a comment - Added content: https://github.com/couchbase/docs/commit/b8094b5c61802fb4b76f0ae6c9f4979c5f1aa6f4
          kzeller kzeller made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          perry Perry Krug added a comment -

          Karen, this isn't just about documentation, this is a bug that needs to be fixed in the product.

          Show
          perry Perry Krug added a comment - Karen, this isn't just about documentation, this is a bug that needs to be fixed in the product.
          perry Perry Krug made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Assignee Karen Zeller [ kzeller ] Aleksey Kondratenko [ alkondratenko ]
          perry Perry Krug made changes -
          Fix Version/s 2.0.2 [ 10418 ]
          Fix Version/s 2.0.1 [ 10399 ]
          kzeller kzeller made changes -
          Component/s documentation [ 10012 ]
          kzeller kzeller made changes -
          Summary [RN 2.0.1] Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment [RN 2.0.2] Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s 2.1 [ 10414 ]
          Fix Version/s 2.0.2 [ 10418 ]
          kzeller kzeller made changes -
          Labels 2.0.1-release-notes 2.0.1-release-notes 2.0.2-release-notes
          Hide
          maria Maria McDuff (Inactive) added a comment -

          deferring out of 2.0.2

          Show
          maria Maria McDuff (Inactive) added a comment - deferring out of 2.0.2
          maria Maria McDuff (Inactive) made changes -
          Affects Version/s 2.0.2 [ 10418 ]
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Labels 2.0.1-release-notes 2.0.2-release-notes ns_server-story
          Hide
          alkondratenko Aleksey Kondratenko (Inactive) added a comment -

          We're simply waiting until we upgrade to R16-something. When that happens this issue will automatically be fixed.

          Show
          alkondratenko Aleksey Kondratenko (Inactive) added a comment - We're simply waiting until we upgrade to R16-something. When that happens this issue will automatically be fixed.
          alkondratenko Aleksey Kondratenko (Inactive) made changes -
          Labels ns_server-story
          Assignee Aleksey Kondratenko [ alkondratenko ] Wayne Siu [ wayne ]
          Component/s installer [ 10221 ]
          Component/s ns_server [ 10019 ]
          Hide
          wayne Wayne Siu added a comment -

          Removing the release notes description in the summary, and assigning it to PM to prioritize

          Show
          wayne Wayne Siu added a comment - Removing the release notes description in the summary, and assigning it to PM to prioritize
          wayne Wayne Siu made changes -
          Assignee Wayne Siu [ wayne ] Anil Kumar [ anil ]
          wayne Wayne Siu made changes -
          Summary [RN 2.0.2] Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment
          maria Maria McDuff (Inactive) made changes -
          Labels scrubbed
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s .next [ 10342 ]
          Fix Version/s 3.0 [ 10414 ]
          maria Maria McDuff (Inactive) made changes -
          Fix Version/s bug-backlog [ 11600 ]
          Fix Version/s feature-backlog [ 10342 ]
          Triage Untriaged [ 10351 ]
          Hide
          schatt Drew Schatt added a comment -

          This is sort of a big issue for us - we were considering deploying couchbase on some rather beefy virtual servers here. In our case, we're using proxmox, and couchbase 2.2.0_community and 2.5.1 both have the issue... which makes them unusable.

          Show
          schatt Drew Schatt added a comment - This is sort of a big issue for us - we were considering deploying couchbase on some rather beefy virtual servers here. In our case, we're using proxmox, and couchbase 2.2.0_community and 2.5.1 both have the issue... which makes them unusable.
          anil Anil Kumar made changes -
          Fix Version/s 3.0 [ 10414 ]
          Fix Version/s bug-backlog [ 11600 ]
          Hide
          anil Anil Kumar added a comment -

          As per Alk with R16 this should be fixed.

          Show
          anil Anil Kumar added a comment - As per Alk with R16 this should be fixed.
          anil Anil Kumar made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          morrie Morrie Schreibman made changes -
          Link This issue relates to CBSE-1428 [ CBSE-1428 ]

            People

            • Assignee:
              anil Anil Kumar
              Reporter:
              creator11 creator11
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes