Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23591

2.5.2 -> 4.6.1 swap rebalance causes empty couchstore files

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Steps

      • Take a 3-node 2.5.2 cluster. One bucket, 30K simple json docs.
      • Rebalance out 1 node
      • Upgrade this node to 4.6.1
      • Swap rebalance back in the upgraded node for one of the 2 remaining 2.5.2 nodes

      On completion of the swap rebalance, several of the couchstore files on the incoming 4.6.1 node contain no documents (they are ~4142 bytes large but only contain the local document, despite the vbucket containing items in memory.

      ls -alS /opt/couchbase/var/lib/couchbase/data/charlie | tail -5
      -rw-rw---- 1 couchbase couchbase  4142 Mar 28 21:43 381.couch.1
      -rw-rw---- 1 couchbase couchbase  4142 Mar 28 21:42 514.couch.1
      -rw-rw---- 1 couchbase couchbase  4142 Mar 28 21:42 527.couch.1
      

      couch_dbdump yields the following:

      # /opt/couchbase/bin/couch_dbdump --local --iterate /opt/couchbase/var/lib/couchbase/data/charlie/381.couch.1
      Dumping "/opt/couchbase/var/lib/couchbase/data/charlie/381.couch.1":
      Key: _local/vbstate
      Value: {"state": "active","checkpoint_id": "2","max_deleted_seqno": "0","failover_table": [{"id":153468350978446,"seq":88},{"id":134463534283871,"seq":0}],"snap_start": "88","snap_end": "88","max_cas": "304708245378"}
       
      Dumping B-tree at 0:
       
      Total docs: 1
      

      Logs
      (node 3 was the 4.6.1 node, being swapped for node2)
      https://cb-engineering.s3.amazonaws.com/davidH/node3-cb252-ubuntu14.zip
      https://cb-engineering.s3.amazonaws.com/davidH/node2-cb252-ubuntu14.zip
      https://cb-engineering.s3.amazonaws.com/davidH/node1-cb252-ubuntu14.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              dhaikney David Haikney (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty