Details

    • Type: Technical task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2011-01-20
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None

      Description

      We need to understand what happens to the cluster if a node runsout of disk space.

      Key questions are:
      1) will the node still service reads
      2) Can the node be failed over
      3) can the node be balanced out
      4) do writes get accepted again once disk space if freed up

      Primary focus should be on Membase nodes, but the experiment should be repeated with memcached buckets (where there is still logging traffic)

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment - - edited

        When a cluster run out of disk space, the node is loaded with data in (node 52) will be death completely.
        http://screencast.com/t/NMwGITI6bz
        Then the death node will automatically reload to initial setup page
        http://screencast.com/t/63vUmCwPxR
        From node 55, we can fail over death node (52) out.
        After fail over, RAM used info on node 55 is incorrect on cluster overview page.
        http://screencast.com/t/m7OlgTNja
        http://screencast.com/t/xQ69bJN9yH
        Delete all data in bucket
        Load data back to bucket (not full)
        Verify data in bucket (ok)
        See RAM quota info incorrectly in data bucket page. File bug MB-3346
        http://screencast.com/t/WcjYDlKC9

        For the death node (52), eventhough it reloads to innitial setup, we can not setup membase server due to no disk space available.
        Stop membase-server service on node 52
        Delete all database file
        Restart membase-server service.
        Go through initial setup
        Load data to default bucket (not full).
        Verify data (ok).
        Add node 55 back to node 52. Rebalance (ok)
        Verify data on both nodes (ok)

        Show
        thuan Thuan Nguyen added a comment - - edited When a cluster run out of disk space, the node is loaded with data in (node 52) will be death completely. http://screencast.com/t/NMwGITI6bz Then the death node will automatically reload to initial setup page http://screencast.com/t/63vUmCwPxR From node 55, we can fail over death node (52) out. After fail over, RAM used info on node 55 is incorrect on cluster overview page. http://screencast.com/t/m7OlgTNja http://screencast.com/t/xQ69bJN9yH Delete all data in bucket Load data back to bucket (not full) Verify data in bucket (ok) See RAM quota info incorrectly in data bucket page. File bug MB-3346 http://screencast.com/t/WcjYDlKC9 For the death node (52), eventhough it reloads to innitial setup, we can not setup membase server due to no disk space available. Stop membase-server service on node 52 Delete all database file Restart membase-server service. Go through initial setup Load data to default bucket (not full). Verify data (ok). Add node 55 back to node 52. Rebalance (ok) Verify data on both nodes (ok)
        Hide
        thuan Thuan Nguyen added a comment -

        I test this case and its behavious is in comment. Sean said we need to raise a warning if disk is almost full.

        Show
        thuan Thuan Nguyen added a comment - I test this case and its behavious is in comment. Sean said we need to raise a warning if disk is almost full.

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            frank Frank Weigel (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes