Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6232

ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (faster with barrier=0)


    • Triage:
    • Is this a Regression?:



      Just try single node dev cluster and try to load beers sample (it'll fail for perhaps different reason).

      You can observe in logs:

      We're doing set_vbucket_state requests from janitor (ns_memcached messages) and this is the first messages from mccouch about just created vbucket (capi_set_view_manager messages and mc_connection messages). Note: I've added extra log messages at the beginning of sync_notify and end to see if it's couchdb's or ns_servers' fault. From timestamps it's clear that it's not.

      [ns_server:debug,2012-08-15T9:47:40.309,n_0@<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [ns_server:info,2012-08-15T9:47:40.309,n_0@<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 740 state to active
      [views:debug,2012-08-15T9:47:40.309,n_0@]Got set_vbucket event for beer-sample/1023. Updated state: active
      [ns_server:info,2012-08-15T9:47:40.309,n_0@<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 739 state to active
      [ns_server:info,2012-08-15T9:47:40.310,n_0@<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 738 state to active
      [ns_server:info,2012-08-15T9:47:40.310,n_0@<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 737 state to active

      And last message is at:

      [ns_server:debug,2012-08-15T9:49:14.410,n_0@<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [views:debug,2012-08-15T9:49:14.410,n_0@]Got set_vbucket event for beer-sample/1. Updated state: active
      [ns_server:debug,2012-08-15T9:49:14.501,n_0@<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
      [views:debug,2012-08-15T9:49:14.502,n_0@]Got set_vbucket event for beer-sample/0. Updated state: active

      I've just retried with remounting with barrier=0 and it was blazing fast. What this mount option changes is short fsync are nearly instant because they merely send data to disk's buffer without waiting until data actually hits platter.

      So that's evidence of some excessive use of fsyncs somewhere during vbucket db files creation and that's clearly on ep-engine/couchstore side.

      Set this to blocker as it'll affect people trying to create buckets on real hardware, hard disks and any modern linux distro (where barriers are on by default).

      1. debug.1.gz
        269 kB
        Tommie McAfee

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.



            • Assignee:
              sundar Sundar Sridharan
              alkondratenko Aleksey Kondratenko (Inactive)
            • Votes:
              0 Vote for this issue
              12 Start watching this issue


              • Created:

                Gerrit Reviews

                There are no open Gerrit changes