Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35332

[Durability]: Rebalance in 2 nodes failed while setup_streams is running for vBuckets. Memcached crashed.

    XMLWordPrintable

Details

    • Triaged
    • No
    • KV-Engine MH Beta part 2

    Description

      Steps to reproduce:
      1. Initialize a 2 node cluster with default bucket having replica=1

      +----------------+----------------------------+--------------+
      		 
      | Nodes          | Services                   | Status       |
      		 
      +----------------+----------------------------+--------------+
      		 
      | 172.23.104.203 | [u'index', u'kv', u'n1ql'] | Cluster node |
      		 
      | 172.23.104.219 | None                       | <--- IN ---  |
      		 
      +----------------+----------------------------+--------------+

      2. Loaded 50k items into the bucket

      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      		 
      | Bucket  | Type    | Replicas | TTL | Items | RAM Quota  | RAM Used  | Disk Used |
      		 
      +---------+---------+----------+-----+-------+------------+-----------+-----------+
      		 
      | default | membase | 1        | 0   | 50000 | 3684696064 | 154512528 | 207095321 |
      		 
      +---------+---------+----------+-----+-------+------------+-----------+-----------+

      3. Rebalance in 2 nodes and load 10k more items in parallel. Lots of DurabilityImpossibleExceptions are seen which is expected as per. Rebalance completed properly.

      +----------------+----------------------------+--------------+
      		 
      | Nodes          | Services                   | Status       |
      		 
      +----------------+----------------------------+--------------+
      		 
      | 172.23.104.203 | [u'index', u'kv', u'n1ql'] | Cluster node |
      		 
      | 172.23.104.219 | [u'kv']                    | Cluster node |
      		 
      | 172.23.104.220 | None                       | <--- IN ---  |
      		 
      | 172.23.104.221 | None                       | <--- IN ---  |
      		 
      +----------------+----------------------------+--------------+

      Rebalance failed, memcached crashed on orchestrator node: ns_1@172.23.104.203

      Attachments

        Issue Links

          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-35334 Snapshot range invariant (snapshot_range_t(a,b) requires start <= end) broken after crash/restart

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty