Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6844

[system test] failed to delete data, cannot locate database file

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0-beta-2
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
    • Environment:
      centos 6.2 64 bit on ec2 build 2.0.0-1808

      Description

      Create a 6 nodes cluster installed with couchbase server 2.0.0-1808. Consistent view is disable.
      Each node has 14 GB RAM and 2 ebs volumes, one for /data and another for /view
      Create 2 bucket and load 9 million items to each bucket.
      Create a doc for bucket.

      When add 2 nodes to cluster, rebalance failed due to one node down (segfault bug MB-6638).
      Stop all loads and restart couchbase server on down node
      During rebalance, got write commit failed error. Check memcached logs, see the following error

      memcached.log.3.txt:Tue Oct 9 00:18:58.901338 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/122.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:18:58.901368 UTC 3: Warning: failed to open database, vbucketId = 122 fileRev = 1 numDocs = 68
      memcached.log.3.txt:Tue Oct 9 00:18:58.901378 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 122 rev = 1
      memcached.log.3.txt:Tue Oct 9 00:18:58.904685 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/55.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:18:58.904700 UTC 3: Warning: failed to open database, vbucketId = 55 fileRev = 1 numDocs = 12
      memcached.log.3.txt:Tue Oct 9 00:18:58.904708 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 55 rev = 1
      memcached.log.3.txt:Tue Oct 9 00:23:24.492253 UTC 3: Warning: couchstore_open_db failed, name=/data/saslbucket/344.couch.1 option=0 rev=1 retried=2 error=no such file [none]
      memcached.log.3.txt:Tue Oct 9 00:23:24.492286 UTC 3: Warning: failed to open database, vbucketId = 344 fileRev = 1 numDocs = 67
      memcached.log.3.txt:Tue Oct 9 00:23:24.492295 UTC 3: Warning: commit failed, cannot save CouchDB docs for vbucket = 344 rev = 1

      memcached.log.3.txt:Tue Oct 9 00:23:40.153322 UTC 3: Warning: failed to delete data, cannot locate database file /data/saslbucket/499.couch.1
      memcached.log.3.txt:Tue Oct 9 00:23:40.153668 UTC 3: Warning: failed to delete data, cannot locate database file /data/saslbucket/499.couch.1

      • Database file exists in node

      [root@ip-10-248-109-239 logs]# ls /data/saslbucket/ | grep 499
      499.couch.1
      [root@ip-10-248-109-239 logs]# ls /data/saslbucket/ | grep 122
      122.couch.1

      Will add collect info later

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #439 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/439/)
        MB-6844 requeue delete operation until the snapshot task is complete (Revision 03360987688eb408bdecefe844b843fb16702e04)
        MB-6844 Increase max retry number for couchstore_open_db (Revision 4f9646ca6ade4fc072165eb97571daf0bde6b9df)

        Result = SUCCESS
        Jin Lim :
        Files :

        • src/ep.cc

        Jin Lim :
        Files :

        • src/couch-kvstore/couch-kvstore.hh
        • src/couch-kvstore/couch-kvstore.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #439 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/439/ ) MB-6844 requeue delete operation until the snapshot task is complete (Revision 03360987688eb408bdecefe844b843fb16702e04) MB-6844 Increase max retry number for couchstore_open_db (Revision 4f9646ca6ade4fc072165eb97571daf0bde6b9df) Result = SUCCESS Jin Lim : Files : src/ep.cc Jin Lim : Files : src/couch-kvstore/couch-kvstore.hh src/couch-kvstore/couch-kvstore.cc
        Hide
        jin Jin Lim (Inactive) added a comment -

        QE tested the toy build with the pending patch set below. No commit failure for delete was found.
        http://review.couchbase.org/#/c/21463/

        Show
        jin Jin Lim (Inactive) added a comment - QE tested the toy build with the pending patch set below. No commit failure for delete was found. http://review.couchbase.org/#/c/21463/
        Hide
        jin Jin Lim (Inactive) added a comment -

        Thanks for the comment. Unfortunately it isn't really easy to distinguish btw persisting write commit failure and temp. write commit failure.
        EP Engine made a modification that increase the number of retry for commit failure that eventually reduce the change of spitting out the false(temp) write commit failure message all over the place. http://review.couchbase.org/#/c/21454/

        Show
        jin Jin Lim (Inactive) added a comment - Thanks for the comment. Unfortunately it isn't really easy to distinguish btw persisting write commit failure and temp. write commit failure. EP Engine made a modification that increase the number of retry for commit failure that eventually reduce the change of spitting out the false(temp) write commit failure message all over the place. http://review.couchbase.org/#/c/21454/
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Jin,

        we need to make that clear for the user so that they dont sense a major issue when they see this error.
        does the current log message in the UI mention which one of these cases we are hitting ?

        if thats the case then we should have this in the 2.0 release notes documentation

        Show
        farshid Farshid Ghods (Inactive) added a comment - Jin, we need to make that clear for the user so that they dont sense a major issue when they see this error. does the current log message in the UI mention which one of these cases we are hitting ? if thats the case then we should have this in the 2.0 release notes documentation
        Hide
        jin Jin Lim (Inactive) added a comment -

        There are two different warnings due to similar Couchstore file i/o access error:
        1) Write commit failure due to open_db error, no such file
        2) Delete data failure due to readdir error, no file is found for the given name

        The first case (1) is a normal behavior under the current design of how the compactor deleting a old db file and renaming a newly compacted db file with a new revision number. There is always a narrow window where CouchKVStore cannot open the old db file while the new db file is being compacted. The retry logic in EP engine takes care of this window. According to the log, the failed writes eventually got succeeded after being re-queued by the retry logic thus no persisting write failures on the given vbuckets.

        The second case is a real issue we need to continue to look at. Please either open a new bug (and close this bug) with the description of (2) or update the current bug accordingly.
        Thanks~

        Show
        jin Jin Lim (Inactive) added a comment - There are two different warnings due to similar Couchstore file i/o access error: 1) Write commit failure due to open_db error, no such file 2) Delete data failure due to readdir error, no file is found for the given name The first case (1) is a normal behavior under the current design of how the compactor deleting a old db file and renaming a newly compacted db file with a new revision number. There is always a narrow window where CouchKVStore cannot open the old db file while the new db file is being compacted. The retry logic in EP engine takes care of this window. According to the log, the failed writes eventually got succeeded after being re-queued by the retry logic thus no persisting write failures on the given vbuckets. The second case is a real issue we need to continue to look at. Please either open a new bug (and close this bug) with the description of (2) or update the current bug accordingly. Thanks~

          People

          • Assignee:
            jin Jin Lim (Inactive)
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes