Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0-beta
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
    • Environment:
      2.0.0-1578, windows

      Description

      Please refer to the screenshot. For build 1578, disk did not drain.

      > Loaded 7M items. Waited for them to be flushed. 3.5M active items stayed in the queue.

      > Memory fragmentation was nearly 2.4G. May relate to disk too.

      Node: 10.2.2.231 (vms)

      Created issue:

      1. cbstats.png
        131 kB
      2. disk_not_drain_windows.png
        69 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        (seemed assigned to wrong folk)

        Show
        steve Steve Yen added a comment - (seemed assigned to wrong folk)
        Hide
        chiyoung Chiyoung Seo added a comment -

        Ronnie,

        When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.

        Show
        chiyoung Chiyoung Seo added a comment - Ronnie, When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.
        Hide
        ronnie Ronnie Sun (Inactive) added a comment -

        attached diag logs

        Show
        ronnie Ronnie Sun (Inactive) added a comment - attached diag logs
        Hide
        ronnie Ronnie Sun (Inactive) added a comment -

        @chiyoung,

        Attached diag logs.

        The cluster is live, 10.2.2.231.

        I'll leave it to you for debugging. Thanks

        Show
        ronnie Ronnie Sun (Inactive) added a comment - @chiyoung, Attached diag logs. The cluster is live, 10.2.2.231. I'll leave it to you for debugging. Thanks
        Hide
        chiyoung Chiyoung Seo added a comment -

        The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all:

        [ns_server:info,2012-08-19T14:27:38.036,ns_1@10.2.2.231:ns_port_memcached:ns_port_server:log:169]memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument

        Show
        chiyoung Chiyoung Seo added a comment - The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all: [ns_server:info,2012-08-19T14:27:38.036,ns_1@10.2.2.231:ns_port_memcached:ns_port_server:log:169] memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument
        Show
        jin Jin Lim added a comment - http://review.couchbase.org/#/c/20109/
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #409 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/)
        MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a)

        Result = SUCCESS
        Jin Lim :
        Files :

        • src/bgfetcher.hh
        • src/flusher.cc
        • src/dispatcher.cc
        • src/dispatcher.hh
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #409 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/ ) MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a) Result = SUCCESS Jin Lim : Files : src/bgfetcher.hh src/flusher.cc src/dispatcher.cc src/dispatcher.hh
        Hide
        chiyoung Chiyoung Seo added a comment -

        This happened it again with the latest fix.

        Show
        chiyoung Chiyoung Seo added a comment - This happened it again with the latest fix.
        Hide
        chiyoung Chiyoung Seo added a comment -

        Ronnie, please attach the diag file.

        Show
        chiyoung Chiyoung Seo added a comment - Ronnie, please attach the diag file.
        Hide
        ronnie Ronnie Sun (Inactive) added a comment -

        Node 10-2-2-232 went down

        Attached diag logs.

        Show
        ronnie Ronnie Sun (Inactive) added a comment - Node 10-2-2-232 went down Attached diag logs.
        Hide
        jin Jin Lim added a comment -

        Please reattache the diag logs. Don't see them. Thanks!

        Show
        jin Jin Lim added a comment - Please reattache the diag logs. Don't see them. Thanks!
        Hide
        jin Jin Lim added a comment -

        Got the logs. Both write commit failure and mutex error occurred again. Looking into it.

        Show
        jin Jin Lim added a comment - Got the logs. Both write commit failure and mutex error occurred again. Looking into it.
        Show
        chiyoung Chiyoung Seo added a comment - http://review.couchbase.org/#/c/20322/
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #419 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/)
        MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48)
        MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884)

        Result = SUCCESS
        Jin Lim :
        Files :

        • src/mutex.cc
        • Makefile.am
        • src/syncobject.hh

        Jin Lim :
        Files :

        • src/couch-kvstore/couch-kvstore.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #419 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/ ) MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48) MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884) Result = SUCCESS Jin Lim : Files : src/mutex.cc Makefile.am src/syncobject.hh Jin Lim : Files : src/couch-kvstore/couch-kvstore.cc
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #421 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/)
        MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf)

        Result = SUCCESS
        Jin Lim :
        Files :

        • src/ep.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #421 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/ ) MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf) Result = SUCCESS Jin Lim : Files : src/ep.cc

          People

          • Assignee:
            jin Jin Lim
            Reporter:
            ronnie Ronnie Sun (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes