Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6844

[system test] failed to delete data, cannot locate database file

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0-beta-2
    • 2.0-beta
    • couchbase-bucket
    • Security Level: Public
    • centos 6.2 64 bit on ec2 build 2.0.0-1808

    Description

      Create a 6 nodes cluster installed with couchbase server 2.0.0-1808. Consistent view is disable.
      Each node has 14 GB RAM and 2 ebs volumes, one for /data and another for /view
      Create 2 bucket and load 9 million items to each bucket.
      Create a doc for bucket.

      When add 2 nodes to cluster, rebalance failed due to one node down (segfault bug MB-6638).
      Stop all loads and restart couchbase server on down node
      During rebalance, got write commit failed error. Check memcached logs, see the following error

      memcached.log.3.txt:Tue Oct 9 00:18:58.901338 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/122.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:18:58.901368 UTC 3: Warning: failed to open database, vbucketId = 122 fileRev = 1 numDocs = 68
      memcached.log.3.txt:Tue Oct 9 00:18:58.901378 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 122 rev = 1
      memcached.log.3.txt:Tue Oct 9 00:18:58.904685 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/55.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:18:58.904700 UTC 3: Warning: failed to open database, vbucketId = 55 fileRev = 1 numDocs = 12
      memcached.log.3.txt:Tue Oct 9 00:18:58.904708 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 55 rev = 1
      memcached.log.3.txt:Tue Oct 9 00:23:24.492253 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/344.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:23:24.492286 UTC 3: Warning: failed to open database, vbucketId = 344 fileRev = 1 numDocs = 67
      memcached.log.3.txt:Tue Oct 9 00:23:24.492295 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 344 rev = 1

      memcached.log.3.txt:Tue Oct 9 00:23:40.153322 UTC 3: Warning: failed to delete data, cannot locate database file /data/saslbucket/499.couch.1
      memcached.log.3.txt:Tue Oct 9 00:23:40.153668 UTC 3: Warning: failed to delete data, cannot locate database file /data/saslbucket/499.couch.1

      • Database file exists in node

      [root@ip-10-248-109-239 logs]# ls /data/saslbucket/ | grep 499
      499.couch.1
      [root@ip-10-248-109-239 logs]# ls /data/saslbucket/ | grep 122
      122.couch.1

      Will add collect info later

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jin Jin Lim (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty