Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57711

Setting num_auxio/nonio_threads=default via REST API hangs memcached

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.2.1
    • 7.1.4, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3
    • memcached
    • Untriaged
    • 0
    • Unknown

    Description

      During testing of MB-56032 it was noted that the AuxIO thread settings API incorrectly interprets "default" if it is used to set the thread count after initial startup - it sets the thread count to 0 (i.e. no AuxIO jobs can run), instead of calculating based on the CPU count.

      Steps to reproduce

      1. Start up a Data Service node.
      2. Change the thread count to "default" 

        curl -u Administrator:asdasd http://localhost:9000/pools/default/settings/memcached/global -d num_auxio_threads=default

      Expected behaviour
      Thread count is set to number of CPUs (8 in this example):

      # ps -eL -q $(pgrep memcached) | grep AuxIo
      		 
        352   394 ?        00:00:00 AuxIoPool0
      		 
        352   395 ?        00:00:00 AuxIoPool1
      		 
        352   396 ?        00:00:00 AuxIoPool2
      		 
        352   397 ?        00:00:00 AuxIoPool3
      		 
        352   398 ?        00:00:00 AuxIoPool4
      		 
        352   399 ?        00:00:00 AuxIoPool5
      		 
        352   400 ?        00:00:00 AuxIoPool6
      		 
        352   401 ?        00:00:00 AuxIoPool7

      Actual behaviour
      Thread count is set to zero - i.e. no threads exist.

      (This actually manifests as one AuxIO thread being left, which is still running the task to update the settings):

          frame #15: 0x000000010553c33c memcached` folly::ThreadPoolExecutor::joinStoppedThreads(this=0x0000000107aaed00, n=<unavailable>)  + 60 at ThreadPoolExecutor.cpp:238
      		 
          frame #16: 0x000000010553bd20 memcached` folly::ThreadPoolExecutor::setNumThreads(this=0x0000000107aaed00, numThreads=<unavailable>)  + 312 at ThreadPoolExecutor.cpp:203
      		 
          frame #17: 0x0000000104ff9404 memcached` std::__1::__function::__func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) [inlined] startExecutorPool(this=<unavailable>, (null)=<unavailable>, s=<unavailable>)::$_72::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) const  + 28 at memcached.cc:754
      		 
          frame #18: 0x0000000104ff93e8 memcached` std::__1::__function::__func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) [inlined] decltype(__f=<unavailable>, __args=<unavailable>, __args=<unavailable>)::$_72&>(fp)(static_cast<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(fp0), static_cast<Settings&>(fp0))) std::__1::__invoke<startExecutorPool()::$_72&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&>(startExecutorPool()::$_72&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)  at type_traits:3918
      		 
          frame #19: 0x0000000104ff93e8 memcached` std::__1::__function::__func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call<startExecutorPool(__args=<unavailable>, __args=<unavailable>, __args=<unavailable>)::$_72&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&>(startExecutorPool()::$_72&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)  at invoke.h:61
      		 
          frame #20: 0x0000000104ff93e8 memcached` std::__1::__function::__func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) [inlined] std::__1::__function::__alloc_func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator(this=<unavailable>, __arg=<unavailable>, __arg=<unavailable>)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)  at function.h:178
      		 
          frame #21: 0x0000000104ff93e8 memcached` std::__1::__function::__func<startExecutorPool()::$_72, std::__1::allocator<startExecutorPool()::$_72>, void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator(this=<unavailable>, __arg=<unavailable>, __arg=<unavailable>)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)  + 32 at function.h:352
      		 
          frame #22: 0x0000000104f7900c memcached` Settings::setNumAuxIoThreads(unsigned long) [inlined] std::__1::__function::__value_func<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator(this=<unavailable>, __args="num_auxio_threads", __args=0x0000000105823090)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) const  + 196 at function.h:505
      		 
          frame #23: 0x0000000104f78ff4 memcached` Settings::setNumAuxIoThreads(unsigned long) [inlined] std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&)>::operator(this=0x0000000107aa6000, __arg="num_auxio_threads", __arg=0x0000000105823090)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Settings&) const  + 4 at function.h:1182
      		 
          frame #24: 0x0000000104f78ff0 memcached` Settings::setNumAuxIoThreads(unsigned long)  + 96 at settings.cc:1303
      		 
          frame #25: 0x0000000104f78f90 memcached` Settings::setNumAuxIoThreads(this=0x0000000105823090, val=<unavailable>)  + 72 at settings.h:780
      		 
          frame #26: 0x0000000104f711d0 memcached` Settings::updateSettings(this=0x0000000105823090, other=0x000000016c1e26c0, apply=<unavailable>)  + 17868 at settings.cc:1239
      		 
          frame #27: 0x0000000104f87420 memcached` SettingsReloadCommandContext::doSettingsReload(this=0x00000001086b81e0)  + 496 at settings_reload_command_context.cc:232
      		 
          frame #28: 0x0000000104f8d6e4 memcached` std::__1::__function::__func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator()() [inlined] SettingsReloadCommandContext::reload(this=<unavailable>)::$_18::operator()() const  + 8 at settings_reload_command_context.cc:285
      		 
          frame #29: 0x0000000104f8d6dc memcached` std::__1::__function::__func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator()() [inlined] decltype(__f=<unavailable>)::$_18&>(fp)()) std::__1::__invoke<SettingsReloadCommandContext::reload()::$_18&>(SettingsReloadCommandContext::reload()::$_18&)  at type_traits:3918
      		 
          frame #30: 0x0000000104f8d6dc memcached` std::__1::__function::__func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator()() [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call<SettingsReloadCommandContext::reload(__args=<unavailable>)::$_18&>(SettingsReloadCommandContext::reload()::$_18&)  at invoke.h:61
      		 
          frame #31: 0x0000000104f8d6dc memcached` std::__1::__function::__func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator()() [inlined] std::__1::__function::__alloc_func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator(this=<unavailable>)()  + 4 at function.h:178
      		 
          frame #32: 0x0000000104f8d6d8 memcached` std::__1::__function::__func<SettingsReloadCommandContext::reload()::$_18, std::__1::allocator<SettingsReloadCommandContext::reload()::$_18>, void ()>::operator(this=<unavailable>)()  + 12 at function.h:352
      		 
          frame #33: 0x0000000104fadec4 memcached` OneShotTask::run() [inlined] std::__1::__function::__value_func<void ()>::operator(this=<unavailable>)() const  + 44 at function.h:505
      		 
          frame #34: 0x0000000104fadeb4 memcached` OneShotTask::run() [inlined] std::__1::function<void ()>::operator(this=0x000000010cc58708)() const  + 4 at function.h:1182
      		 
          frame #35: 0x0000000104fadeb0 memcached` OneShotTask::run(this=0x000000010cc58698)  + 24 at one_shot_task.h:50
      		 
          frame #36: 0x00000001054099fc memcached` GlobalTask::execute(this=0x000000010cc58698, threadName=<unavailable>)  + 176 at globaltask.cc:98

      The same issue occurs with NonIO threads - "default" results in zero being created.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. MB-56032 Increase default number of KV-Engine AuxIO threads

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-49977 Cannot make persistent change to num nonio/auxio threads

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-57711
          # Subject Branch Project Status CR V
          193576,3 MB-57711 / MB-52290: Add ability to get thread counts from memcached stats neo kv_engine Status: MERGED +2 +1
          193577,2 MB-57711: report both configured and actual thread stats neo kv_engine Status: MERGED +2 +1
          193578,2 MB-57711: Fix calculation of num_auxio_threads=default neo kv_engine Status: MERGED +2 +1
          193579,3 MB-57711: Fix calculation of num_nonio_threads=default neo kv_engine Status: MERGED +2 +1
          196477,1 Merge commit '030595f79' into 'couchbase/master' master kv_engine Status: MERGED +2 +1
          196478,1 Merge commit '67995e151' into 'couchbase/master' master kv_engine Status: ABANDONED 0 0
          196598,3 Merge commit '67995e151' into couchbase/master master kv_engine Status: MERGED +2 +1
          196601,3 Merge commit '5b3d16710' into couchbase/master master kv_engine Status: MERGED +2 +1
          196607,1 Merge commit 'fffa122aa' into couchbase/master master kv_engine Status: MERGED +2 +1

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty