Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6232

ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (faster with barrier=0)

    Details

    • Triage:
      Triaged
    • Is this a Regression?:
      Yes

      Description

      SUBJ.

      Just try single node dev cluster and try to load beers sample (it'll fail for perhaps different reason).

      You can observe in logs:

      We're doing set_vbucket_state requests from janitor (ns_memcached messages) and this is the first messages from mccouch about just created vbucket (capi_set_view_manager messages and mc_connection messages). Note: I've added extra log messages at the beginning of sync_notify and end to see if it's couchdb's or ns_servers' fault. From timestamps it's clear that it's not.

      [ns_server:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [ns_server:info,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 740 state to active
      [views:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1023. Updated state: active
      [ns_server:info,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 739 state to active
      [ns_server:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done
      [ns_server:info,2012-08-15T9:47:40.310,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 738 state to active
      [ns_server:info,2012-08-15T9:47:40.310,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 737 state to active

      And last message is at:

      [ns_server:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [views:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1. Updated state: active
      [ns_server:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done
      [ns_server:debug,2012-08-15T9:49:14.501,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [views:debug,2012-08-15T9:49:14.502,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/0. Updated state: active
      [ns_server:debug,2012-08-15T9:49:14.502,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done

      I've just retried with remounting with barrier=0 and it was blazing fast. What this mount option changes is short fsync are nearly instant because they merely send data to disk's buffer without waiting until data actually hits platter.

      So that's evidence of some excessive use of fsyncs somewhere during vbucket db files creation and that's clearly on ep-engine/couchstore side.

      Set this to blocker as it'll affect people trying to create buckets on real hardware, hard disks and any modern linux distro (where barriers are on by default).

      1. debug.1.gz
        269 kB
        Tommie McAfee

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          chiyoung Chiyoung Seo added a comment -

          There is mis-communication among the teams. We identified the root cause and still need to fix this issue, but not for 3.0 as it requires changing a way of storing vbucket_state docs and creating individual vbucket database files during the bucket instantiation.

          Show
          chiyoung Chiyoung Seo added a comment - There is mis-communication among the teams. We identified the root cause and still need to fix this issue, but not for 3.0 as it requires changing a way of storing vbucket_state docs and creating individual vbucket database files during the bucket instantiation.
          Hide
          ingenthr Matt Ingenthron added a comment -

          Chiyoung: who is the appropriate owner for this?

          Show
          ingenthr Matt Ingenthron added a comment - Chiyoung: who is the appropriate owner for this?
          Hide
          chiyoung Chiyoung Seo added a comment -

          Sundar still owns this ticket. I already communicated with him regarding how we can resolve this issue.

          Show
          chiyoung Chiyoung Seo added a comment - Sundar still owns this ticket. I already communicated with him regarding how we can resolve this issue.
          Hide
          wayne Wayne Siu added a comment -

          Triaged in Maintenance Meeting on Dec 16, 2014.
          Updating the affected and fixed versions.

          Show
          wayne Wayne Siu added a comment - Triaged in Maintenance Meeting on Dec 16, 2014. Updating the affected and fixed versions.
          Hide
          chiyoung Chiyoung Seo added a comment -

          I don't think this can be addressed by a sherlock release. When any long-lasting issue needs to be moved to the next major release, please communicate and discuss it with the corresponding component engineers.

          Show
          chiyoung Chiyoung Seo added a comment - I don't think this can be addressed by a sherlock release. When any long-lasting issue needs to be moved to the next major release, please communicate and discuss it with the corresponding component engineers.

            People

            • Assignee:
              sundar Sundar Sridharan
              Reporter:
              alkondratenko Aleksey Kondratenko (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:

                Gerrit Reviews

                There are no open Gerrit changes