Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10832

cbrestore sometimes fails or hangs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 3.0
    • 2.2.0
    • tools
    • Security Level: Public
    • None
    • This is on a VM with the following:
      - 1 VCPU
      - 2GB memory
      - 20GB root disk
    • Untriaged
    • Unknown

    Description

      Based on the environment (above), I have 5 couchbase buckets (each with ~2000 documents) + 2 memcached buckets (with no data).

      Here are the steps I performed:

      1) on VM1:
      $ curl -u root:password http://localhost:8091/pools/default/buckets > /tmp/buckets.json
      $ /opt/couchbase/bin/cbbackup http://localhost:8091 /tmp/backups -u root -p password
      $ mv /tmp/buckets.json /tmp/backups
      $ ls /tmp/backups/*
      /tmp/backups/buckets.json

      /tmp/backups/bucket-michael1:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-michael2:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-michael3:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-test1:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-test2:
      design.json node-127.0.0.1%3A8091

      2) created VM2 with the same environment as VM1

      3) on VM2:
      $ scp -r VM1:/tmp/backups /tmp

      $ ls /tmp/backups/*
      /tmp/backups/buckets.json

      /tmp/backups/bucket-michael1:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-michael2:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-michael3:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-test1:
      design.json node-127.0.0.1%3A8091

      /tmp/backups/bucket-test2:
      design.json node-127.0.0.1%3A8091

      1. based on /tmp/backups/buckets.json... for each bucket:
      2. 1. create the bucket
      3. 2. if couchbase bucket, wait for bucket to finish creating
      4. 3. if couchbase bucket, restore
      1. create couchbase bucket
        $ curl -X POST -u root:password -d name="michael1" -d bucketType="couchbase" -d ramQuotaMB="100" -d authType="sasl" -d saslPassword="password" -d proxyPort="0" -d replicaNumber="1" -d replicaIndex="1" -d threadsNumber="2" -d flushEnabled="1" http://localhost:8091/pools/default/buckets
      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/michael1
      1. restore
        $ /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=michael1 --bucket-destination=michael1 -u root -p password
      1. create couchbase bucket
        $ curl -X POST -u root:password -d name="michael2" -d bucketType="couchbase" -d ramQuotaMB="100" -d authType="sasl" -d saslPassword="password" -d proxyPort="0" -d replicaNumber="0" -d replicaIndex="0" -d threadsNumber="2" -d flushEnabled="0" http://localhost:8091/pools/default/buckets
      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/michael2
      1. restore
        $ /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=michael2 --bucket-destination=michael2 -u root -p password
      1. create couchbase bucket
        $ curl -X POST -u root:password -d name="michael3" -d bucketType="couchbase" -d ramQuotaMB="100" -d authType="sasl" -d saslPassword="password" -d proxyPort="0" -d replicaNumber="1" -d replicaIndex="0" -d threadsNumber="2" -d flushEnabled="0" http://localhost:8091/pools/default/buckets
      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/michael3
      1. restore
        $ /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=michael3 --bucket-destination=michael3 -u root -p password
      1. create memcached buckets
        $ curl -X POST -u root:password -d name="sample1" -d bucketType="memcached" -d ramQuotaMB="100" -d authType="sasl" -d saslPassword="password" -d proxyPort="0" -d replicaNumber="0" -d replicaIndex="1" -d threadsNumber="3" -d flushEnabled="1" http://localhost:8091/pools/default/buckets
        $ curl -X POST -u root:password -d name="sample2" -d bucketType="memcached" -d ramQuotaMB="100" -d authType="none" -d saslPassword="" -d proxyPort="11214" -d replicaNumber="0" -d replicaIndex="1" -d threadsNumber="3" -d flushEnabled="0" http://localhost:8091/pools/default/buckets
      1. create couchbase bucket
        $ curl -X POST -u root:password -d name="test1" -d bucketType="couchbase" -d ramQuotaMB="100" -d authType="sasl" -d saslPassword="password" -d proxyPort="0" -d replicaNumber="1" -d replicaIndex="1" -d threadsNumber="2" -d flushEnabled="1" http://localhost:8091/pools/default/buckets
      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/test1
      1. restore
        $ /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=test1 --bucket-destination=test1 -u root -p password
      1. create couchbase bucket
        $ curl -X POST -u root:password -d name="test2" -d bucketType="couchbase" -d ramQuotaMB="100" -d authType="none" -d saslPassword="" -d proxyPort="11212" -d replicaNumber="0" -d replicaIndex="0" -d threadsNumber="2" -d flushEnabled="0" http://localhost:8091/pools/default/buckets
      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/test2
      1. restore
        $ /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=test2 --bucket-destination=test2 -u root -p password

      ======== ERRORS ==============
      During these steps, sometimes I would notice the following errors with cbrestore:
      1) Unexpected JSON value
      Unexpected error while running command.
      Command: /opt/couchbase/bin/cbrestore /tmp/backups couchbase://localhost:8091 --bucket-source=test2 --bucket-destination=test2 -u root -p Fwg3jVkArvQsG2RqaNxPUDeZFNB28Uynnxxy
      Exit code: 1
      Stdout: ''
      Stderr: '2014-04-09 20:56:25,932: mt skipping bucket that is not a couchbase-bucket: sample1\n2014-04-09 20:56:25,960: mt skipping bucket that is not a couchbase-bucket: sample2\nerror: unexpected JSON value from: http://localhost:8091 - did you provide the right source URL?\n'

      2) Hangs (I have it timeout after 10 minutes)
      Time out after waiting 600 seconds when running proc: ('/opt/couchbase/bin/cbrestore', '/tmp/backups', 'couchbase://localhost:8091', u'-bucket-source=test2', u'-bucket-destination=test2', '-u', 'root', '-p', 'Fwg3jVkArvQsG2RqaNxPUDeZFNB28Uynnxxy')

      ======== QUESTIONS ==============
      1) Why is cbrestore inconsistent? Somtimes, it succeeds. Somtimes, it fails.

      2) Does it have to do with the bucket not finish creating yet? Is there a better way to verify the bucket finished creating other than the approach I took (see below)?

      1. wait for bucket to finish creating (look for vBucketServerMap -> vBucketMap)
        $ curl -u root:password http://localhost:8091/pools/default/buckets/test1

      3) Is there a suggested amount of time to wait after creating the bucket before using cbrestore to restore?

      Attachments

        1. babysitter.1
          52 kB
        2. bucket.py
          0.8 kB
        3. buckets.json
          45 kB
        4. debug.1
          10.00 MB
        5. info.1
          1.03 MB
        6. recreate-2014-04-15.txt
          4 kB
        7. recreate-2014-04-17.txt
          4 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            bcui Bin Cui (Inactive)
            michayu michayu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty