Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4735

Data loss during rebalance after service restart

    Details

      Description

      • start development cluster of two nodes
      • set up first node with default bucket
      • join second node but don't rebalance yet
      • load some data
      • wait till it hits disk
      • Ctrl-C and start this cluster back
      • wait till warmup is done and observe that data is there
      • run rebalance
      • observe that after rebalance new node has zero items. Those are lost.

      It's not happening if we don't restart. And same happens if I just kill -9 memcached instead of restarting everything.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        Chiyoung, this showed up at the recent sprint demo... Can you analyze?

        Show
        steve Steve Yen added a comment - Chiyoung, this showed up at the recent sprint demo... Can you analyze?
        Hide
        keith Keith Batten (Inactive) added a comment -

        this is blocking warmup performance testing as well

        Show
        keith Keith Batten (Inactive) added a comment - this is blocking warmup performance testing as well
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        just to make it a bit more clear. The data is there after restart. It's just somehow not there after rebalance.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - just to make it a bit more clear. The data is there after restart. It's just somehow not there after rebalance.
        Hide
        keith Keith Batten (Inactive) added a comment -

        ok, i'll verify if i see this in my testing if i do a full service restart instead of just restarting memcached. if the data warms up properly with a service restart then i won't be blocked.

        Show
        keith Keith Batten (Inactive) added a comment - ok, i'll verify if i see this in my testing if i do a full service restart instead of just restarting memcached. if the data warms up properly with a service restart then i won't be blocked.
        Hide
        steve Steve Yen added a comment -

        Chiyoung, any update on this? Thx

        Show
        steve Steve Yen added a comment - Chiyoung, any update on this? Thx
        Hide
        chiyoung Chiyoung Seo added a comment -

        This issue is a duplicate of http://www.couchbase.com/issues/browse/MB-4595.

        It's a edge case that can only happen when all the nodes have the open checkpoint with id 1 and are restarted. I don't think this is a blocker, but will fix it for DP4 and 1.8.1.

        Show
        chiyoung Chiyoung Seo added a comment - This issue is a duplicate of http://www.couchbase.com/issues/browse/MB-4595 . It's a edge case that can only happen when all the nodes have the open checkpoint with id 1 and are restarted. I don't think this is a blocker, but will fix it for DP4 and 1.8.1.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Dupe.

        Show
        maria Maria McDuff (Inactive) added a comment - Dupe.

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            alkondratenko Aleksey Kondratenko (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes