Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4006

mb_mnesia,ensure_schema timeout when restarting ns_server

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Incomplete
    • Affects Version/s: 1.7.0, 1.7.2
    • Fix Version/s: None
    • Component/s: ns_server
    • Security Level: Public
    • Environment:
      1.7 GA

      Description

      users are run into this issue too often which i suspect has sth to do with the uptime or maybe an unclean shutdown of the server can result in this behavior.

      asking customers to delete their statistics is not a very pleasent thing to do

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        farshid Farshid Ghods (Inactive) created issue -
        farshid Farshid Ghods (Inactive) made changes -
        Field Original Value New Value
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        When a process is killed or dies, mnesia files might be left in a poor state that prevents restart.

        Show
        farshid Farshid Ghods (Inactive) added a comment - When a process is killed or dies, mnesia files might be left in a poor state that prevents restart.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        the workaround is to remove all files under /opt/membase/var/lib/membase/mnesia/.

        Show
        farshid Farshid Ghods (Inactive) added a comment - the workaround is to remove all files under /opt/membase/var/lib/membase/mnesia/ .
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        fix merged

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - fix merged
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        farshid Farshid Ghods (Inactive) made changes -
        Labels 1.7.0-release-notes 1.7.0-release-notes 1.8.0-release-notes
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        reported by a user on 1.7.2
        diags here : http://www63.zippyshare.com/v/19241756/file.html

        Show
        farshid Farshid Ghods (Inactive) added a comment - reported by a user on 1.7.2 diags here : http://www63.zippyshare.com/v/19241756/file.html
        farshid Farshid Ghods (Inactive) made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        ingenthr Matt Ingenthron made changes -
        Fix Version/s 1.7.1 [ 10183 ]
        Affects Version/s 1.7.2 [ 10203 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        thanks for raising this again.

        Looks like our workaround is not working all the time. For now manual workaround of deleting mnesia files and starting ns_server again will work.

        For future we'll get rid of mnesia completely.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - thanks for raising this again. Looks like our workaround is not working all the time. For now manual workaround of deleting mnesia files and starting ns_server again will work. For future we'll get rid of mnesia completely.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Dipti, maybe I'm missing something and stats-in-mnesia is not "unfixable crap". But my plan since before 1.7.0 was to replace saving stats in mnesia with much simpler periodic snapshotting into plain file. Because stats are not really precious thing.

        I estimate this work to be around 1-2 days including thorough testing.

        We also had some plans for storing stats in couch, but given KISS principle I think my simplified approach is what we need.

        Please, decide when we want this work to be done.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Dipti, maybe I'm missing something and stats-in-mnesia is not "unfixable crap". But my plan since before 1.7.0 was to replace saving stats in mnesia with much simpler periodic snapshotting into plain file. Because stats are not really precious thing. I estimate this work to be around 1-2 days including thorough testing. We also had some plans for storing stats in couch, but given KISS principle I think my simplified approach is what we need. Please, decide when we want this work to be done.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Dipti Borkar [ dipti ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Let me explain "not really precious thing" better. Apparently it's too easily to misunderstand my original phrase as "we shouldn't save stats at all".

        Mnesia does it's own journalling and crash recovery and durable commits and stuff like that. Or supposedly does.

        My point was we don't need that level of durability. That means that instead of trying to commit supposedly durable transaction via mnesia after each stats sample we'll accumulate changes and save them once or twice per minute. Saving itself (because it's not much data) can be made durable and cheap.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Let me explain "not really precious thing" better. Apparently it's too easily to misunderstand my original phrase as "we shouldn't save stats at all". Mnesia does it's own journalling and crash recovery and durable commits and stuff like that. Or supposedly does. My point was we don't need that level of durability. That means that instead of trying to commit supposedly durable transaction via mnesia after each stats sample we'll accumulate changes and save them once or twice per minute. Saving itself (because it's not much data) can be made durable and cheap.
        Hide
        dc Daniel added a comment -

        Attaching the relevant logs.

        Show
        dc Daniel added a comment - Attaching the relevant logs.
        dc Daniel made changes -
        Attachment log.rar [ 12985 ]
        peter peter made changes -
        Comment [ Lu provided by the 09 series in a warm summer morning coastline, <strong><a href="http://www.replicawatchesale1.com/">Replica Rolex</a></strong> think. Surprisingly, the temperature, one mile one mile natural sand, <strong><a href="http://www.popreplicabags.com/">Replica Handbags</a></strong> part of the vast expanse of water, heat the coast of the pie, Ipanema beach on the beautiful landscape diversity. We will conduct our bathing suits, sunglasses, bags, bracelets, links ... <strong><a href="http://www.popreplicabags.com/">Louis Vuitton Handbags</a></strong> whatever you bring you to enjoy your glorious period. Of course, <strong><a href="http://www.popreplicabags.com/louis-vuitton-handbags-louis-vuitton-speedy-c-1_2_33.html">Louis Vuitton Speedy</a></strong> the most important commodity on the beach, so the need for frequent bag, we can easily invest in those clothes, as well as with spicy each of us demonstrators, Cabas the same time.Your of Ipanema Beach is the main function. very suitable for people who are looking for an exciting beach holiday. The popular and stylish, and definitely should never be a magnificent coastline. As the hot visual appeal, <strong><a href="http://www.popreplicabags.com/louis-vuitton-handbags-louis-vuitton-neverfull-c-1_2_34.html">Louis Vuitton Neverfull</a></strong> my wife and I almost Tahitienne series supervision concept! <strong><a href="http://www.popreplicabags.com/louis-vuitton-handbags-louis-vuitton-alma-c-1_2_35.html">Louis Vuitton Alma</a></strong> Seriously with these people may be very similar, while the spherical design.Take Beach Cabas Ipanema, <strong><a href="http://www.popreplicabags.com/louis-vuitton-handbags-louis-vuitton-artsy-c-1_2_36.html">Louis Vuitton Artsy</a></strong> a good form of advertising on the Internet. It's just an elegant open-air use of organic cotton bag material, logo, excellent color, fine pleated, tapered basic structure of the management of every detail, <strong><a href="http://www.popreplicabags.com/louis-vuitton-handbags-louis-vuitton-keepall-c-1_2_37.html">Louis Vuitton Keepall</a></strong> in full compliance with the occasion of your beach. ]
        peter peter made changes -
        Fix Version/s .next [ 10342 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Invalidated by removal of mnesia

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Invalidated by removal of mnesia
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Assignee Dipti Borkar [ dipti ] Aleksey Kondratenko [ alkondratenko ]
        Fix Version/s .next [ 10342 ]
        Resolution Incomplete [ 4 ]
        Hide
        dipti Dipti Borkar added a comment -

        have you already removed mnesia?

        If so is there any impact in the upgrade process ?

        Show
        dipti Dipti Borkar added a comment - have you already removed mnesia? If so is there any impact in the upgrade process ?
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        The code was actually ready for 2.0.1 but sadly too late. So 2.0.2 got it early.

        Impact on upgrade process is simple. People will lose their stats archive. I.e. we don't bother converting stats from mnesia to new format. But note that rolling upgrade already causes them to lose historical stats.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - The code was actually ready for 2.0.1 but sadly too late. So 2.0.2 got it early. Impact on upgrade process is simple. People will lose their stats archive. I.e. we don't bother converting stats from mnesia to new format. But note that rolling upgrade already causes them to lose historical stats.

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes