Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6662

XDC: worse performance with expired items

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: XDCR
    • Security Level: Public
    • Labels:
    • Environment:
      VMs, CentOS 6.2, 4-to-4 nodes, build 1723

      Description

      Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes):
      – extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
      – XDC queue grows all the time on source side (unlike case w/o expired items)

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #431 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/)
        MB-6662 Ignore the item's expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59)

        Result = SUCCESS
        Chiyoung Seo :
        Files :

        • src/stored-value.hh
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #431 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/ ) MB-6662 Ignore the item's expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59) Result = SUCCESS Chiyoung Seo : Files : src/stored-value.hh
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Verified.

        Show
        pavelpaulau Pavel Paulau added a comment - Verified.
        Show
        chiyoung Chiyoung Seo added a comment - http://review.couchbase.org/#/c/20927/1
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops.

        I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section.

        Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190

        By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely.

        Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
        ep_num_ops_del_meta: 34329
        ep_num_ops_get_meta: 41910986
        ep_num_ops_set_meta: 14257849
        Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
        ep_num_ops_del_meta: 34329
        ep_num_ops_get_meta: 44140162
        ep_num_ops_set_meta: 14257849
        Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
        ep_num_ops_del_meta: 34329
        ep_num_ops_get_meta: 44226426
        ep_num_ops_set_meta: 14257849
        Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
        ep_num_ops_del_meta: 34329
        ep_num_ops_get_meta: 44254887
        ep_num_ops_set_meta: 14257849
        Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
        ep_num_ops_del_meta: 34329
        ep_num_ops_get_meta: 44273589
        ep_num_ops_set_meta: 14257849

        Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation.

        Thanks.

        Show
        junyi Junyi Xie (Inactive) added a comment - OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops. I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section. Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190 By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely. Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta ep_num_ops_del_meta: 34329 ep_num_ops_get_meta: 41910986 ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta ep_num_ops_del_meta: 34329 ep_num_ops_get_meta: 44140162 ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta ep_num_ops_del_meta: 34329 ep_num_ops_get_meta: 44226426 ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta ep_num_ops_del_meta: 34329 ep_num_ops_get_meta: 44254887 ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta ep_num_ops_del_meta: 34329 ep_num_ops_get_meta: 44273589 ep_num_ops_set_meta: 14257849 Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation. Thanks.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Tried build 1723 w/o expired items and everything looks fine (at least as it was before).

        Show
        pavelpaulau Pavel Paulau added a comment - Tried build 1723 w/o expired items and everything looks fine (at least as it was before).
        Hide
        ketaki Ketaki Gangal added a comment -

        Closed MB-6563, using this bug to track expired items.

        Show
        ketaki Ketaki Gangal added a comment - Closed MB-6563 , using this bug to track expired items.
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        Known issue. Duplicate of MB-6563.

        Show
        junyi Junyi Xie (Inactive) added a comment - Known issue. Duplicate of MB-6563 .
        Hide
        pavelpaulau Pavel Paulau added a comment -

        jic: reports.

        Show
        pavelpaulau Pavel Paulau added a comment - jic: reports.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Ketaki,
        Please update/close your bug and assign this issue to Junyi.

        http://www.couchbase.com/issues/browse/MB-6563

        Show
        pavelpaulau Pavel Paulau added a comment - Ketaki, Please update/close your bug and assign this issue to Junyi. http://www.couchbase.com/issues/browse/MB-6563

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes