Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6342

User-triggered compaction of view `default/_design/dev_view4` failed: shutdown, when rebalancing out a server node

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0
    • Component/s: ns_server, view-engine
    • Security Level: Public
    • Labels:
      None
    • Environment:
      CentOS 64 bit

      Description

      1) Swap rebalance one node in and out
      2) Trigger compaction for production view

      We see the following error on all the nodes in the cluster:-
      User-triggered compaction of view `default/_design/dev_view4` failed: shutdown. See logs for detailed reason.

      We should probably ignore bucket shutdown error.
      Along the same lines as
      http://www.couchbase.com/issues/browse/MB-6278

      [ns_server:warn,2012-08-21T0:57:56.407,ns_1@10.3.121.16:<0.15895.9>:compaction_daemon:do_chain_compactors:520]Compactor for view `default-s/_design/dev_view1` (pid [

      {type,view},
      {name, <<"default-s/_design/dev_view1">>},
      {important,false},
      {fa,
      {#Fun<compaction_daemon.21.91655267>,
      [<<"default-s">>,
      <<"_design/dev_view1">>,
      {config,
      {30, 18446744073709551616},
      {30, 18446744073709551616},
      undefined,false,
      {daemon_config,30,
      131072}},
      false,bucket]}}]) terminated unexpectedly (ignoring this): {updater_died,
      {updater_error,
      shutdown}}


      [ns_server:warn,2012-08-21T0:17:30.366,ns_1@10.3.121.16:<0.15369.9>:compaction_daemon:do_chain_compactors:515]Compactor for view `default/_design/dev_view4/main` (pid [{type,view}

      ,

      {important,true}

      ,

      {name, <<"default/_design/dev_view4/main">>}

      ,
      {fa,
      {#Fun<compaction_daemon.16.5288869>,
      [<<"default">>,
      <<"_design/dev_view4">>,
      main,
      {config,

      {30, 18446744073709551616},
      {30, 18446744073709551616}

      ,
      undefined,false,
      {daemon_config,
      30,131072}},
      true,

      {view, <<"_design/dev_view4">>}

      ]}}]) terminated unexpectedly: shutdown

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        karan Karan Kumar (Inactive) created issue -
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        This is normal to happen during rebalance and can safelly be ignored.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - This is normal to happen during rebalance and can safelly be ignored.
        FilipeManana Filipe Manana (Inactive) made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Assignee Aliaksey Artamonau [ aliaksey artamonau ] Filipe Manana [ filipemanana ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Component/s 3rd-party [ 10219 ]
        Component/s ns_server [ 10019 ]
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        At The very best, ns_ server compaction daemon can ignore this particular exit reason and retry immediately or leave it for next iteration.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - At The very best, ns_ server compaction daemon can ignore this particular exit reason and retry immediately or leave it for next iteration.
        FilipeManana Filipe Manana (Inactive) made changes -
        Component/s ns_server [ 10019 ]
        Component/s view-engine [ 10060 ]
        Component/s 3rd-party [ 10219 ]
        Hide
        Aliaksey Artamonau Aliaksey Artamonau added a comment -

        The problem with user-triggered compactions here is that in case something goes wrong we show the error in the UI log. So it might be confusing for the user to see this kind of errors. But it's not trivial to ignore those because the actual error can look differently. For instance, in one case above it's just 'shutdown', in the other {updater_died, {updater_error, shutdown}}. There can be other variants probably.

        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - The problem with user-triggered compactions here is that in case something goes wrong we show the error in the UI log. So it might be confusing for the user to see this kind of errors. But it's not trivial to ignore those because the actual error can look differently. For instance, in one case above it's just 'shutdown', in the other {updater_died, {updater_error, shutdown}}. There can be other variants probably.
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Aliaksey, the only expected, harmless exit reasons, are:

        shutdown

        {updater_died, shutdown} {updater_died, noproc}
        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Aliaksey, the only expected, harmless exit reasons, are: shutdown {updater_died, shutdown} {updater_died, noproc}
        Aliaksey Artamonau Aliaksey Artamonau made changes -
        Assignee Filipe Manana [ filipemanana ] Aliaksey Artamonau [ aliaksey artamonau ]
        Aliaksey Artamonau Aliaksey Artamonau made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Thanks. I would believe these would still get logged in ns_server logs and be not UI visible?

        Show
        karan Karan Kumar (Inactive) added a comment - Thanks. I would believe these would still get logged in ns_server logs and be not UI visible?
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #448 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/448/)
        MB-6342 Restart view compaction when getting shutdown errors. (Revision eed96a45ef4182debd1a3d5d8b9550f8e904b0b6)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/compaction_daemon.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #448 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/448/ ) MB-6342 Restart view compaction when getting shutdown errors. (Revision eed96a45ef4182debd1a3d5d8b9550f8e904b0b6) Result = SUCCESS Aliaksey Kandratsenka : Files : src/compaction_daemon.erl
        Aliaksey Artamonau Aliaksey Artamonau made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Aliaksey Artamonau Aliaksey Artamonau
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes