Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7285

Occasional badmatch in index updater if it performs cleanup (was [system test] view manual compaction failed with error "updater_died, {updater_error, {badmatch, {ok, <<0,0,58,)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0
    • Component/s: view-engine
    • Security Level: Public
    • Labels:
    • Environment:
      windows 2008 R2 64 bit build 2.0.0-1965

      Description

      Do system test on windows in ec2
      Environment: Windows 2008 R2 64bit in EC2
      Total server: 8
      Cluster info:

      • 6 nodes cluster
      • one default bucket with size 12GB RAM per node
      • Load 48 million items to cluster with size from 128 to 512 byte.
      • Create one doc with 2 views

      Do add in, remove and failover test.

      During rebalance, trigger manually view compaction by click on compact button in view section. Check log and see compaction failed with error: "User-triggered compaction of view `default/_design/d1` failed: {updater_died,
      {updater_error,"

      Look in diags of node 10.224.47.35, I see these error

      [user:error,2012-11-28T17:40:16.115,ns_1@10.224.47.35:compaction_daemon<0.26297.30>:compaction_daemon:handle_info:324]User-triggered compaction of view `default/_design/d1` failed: {updater_died,
      {updater_error,
      {badmatch,
      {ok,
      <<0,0,58,
      58,118,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      12,0,
      240,0,0,
      0,0,0,0,
      0,0,0,0,
      0,3,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,252,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,1,
      255,252,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,255,
      255,248,
      0,0,0,0,
      0,0,0,0,
      0,127,
      255,255,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      1,127,
      255,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,3,
      254,0>>}}}}. See logs for detailed reason.
      [ns_server:warn,2012-11-28T17:40:16.115,ns_1@10.224.47.35:<0.28230.107>:compaction_daemon:do_chain_compactors:514]Compactor for view `default/_design/d1/replica` (pid [

      {type,view},
      {important,true},
      {name, <<"default/_design/d1/replica">>},
      {fa,
      {#Fun<compaction_daemon.16.64676466>,
      [<<"default">>,
      <<"_design/d1">>,
      replica,
      {config,
      {30, 18446744073709551616},
      {30, 18446744073709551616},
      undefined,false,
      {daemon_config,30,
      131072}},
      false,
      {[{type,bucket}]}]}}]) terminated unexpectedly: {updater_died,
      {updater_error,
      {badmatch,
      {ok,
      <<0,0,58,
      58,118,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      12,0,
      240,0,0,
      0,0,0,0,
      0,0,0,0,
      0,3,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,252,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,1,
      255,252,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      0,255,
      255,248,
      0,0,0,0,
      0,0,0,0,
      0,127,
      255,255,
      0,0,0,0,
      0,0,0,0,
      0,0,0,0,
      1,127,
      255,0,0,
      0,0,0,0,
      0,0,0,0,
      0,0,3,
      254,0>>}}}}
      [ns_server:warn,2012-11-28T17:40:16.130,ns_1@10.224.47.35:<0.13946.106>:compaction_daemon:do_chain_compactors:519]Compactor for view `default/_design/d1` (pid [{type,view}

      ,

      And few more crashes report

      =========================CRASH REPORT=========================
      crasher:
      initial call: compaction_daemon:spawn_view_index_compactor/6-fun-0/0
      pid: <0.4216.108>
      registered_name: []
      exception exit: {updater_died,
      {updater_error,
      {badmatch,
      {ok,<<0,0,58,58,118,0,0,0,0,0,0,0,0,0,0,0,
      0,12,0,240,0,0,0,0,0,0,0,0,0,0,0,3,0,
      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      1,255,252,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,0,0,255,255,248,0,0,0,0,0,0,0,0,
      0,127,255,255,0,0,0,0,0,0,0,0,0,0,0,
      0,1,127,255,0,0,0,0,0,0,0,0,0,0,0,0,
      3,254,0>>}}}}
      in function compaction_daemon:do_spawn_view_index_compactor/5
      in call from compaction_daemon:'spawn_view_index_compactor/6-fun-0'/7
      ancestors: [<0.28230.107>,<0.13946.106>,<0.13945.106>,compaction_daemon,
      <0.26296.30>,ns_server_sup,ns_server_cluster_sup,<0.67.0>]
      messages: []
      links: [<0.28230.107>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 4181
      stack_size: 24
      reductions: 2704
      neighbours:

      [ns_server:debug,2012-11-28T17:40:16.130,ns_1@10.224.47.35:compaction_daemon<0.26297.30>:compaction_daemon:handle_info:269]Starting compaction for the following buckets:
      [<<"default">>]
      [error_logger:error,2012-11-28T17:40:16.130,ns_1@10.224.47.35:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: compaction_daemon:spawn_view_index_compactor/6-fun-0/0
      pid: <0.4215.108>
      registered_name: []
      exception exit: {updater_died,
      {updater_error,
      {badmatch,
      {ok,<<0,0,58,58,118,0,0,0,0,0,0,0,0,0,0,0,
      0,12,0,240,0,0,0,0,0,0,0,0,0,0,0,3,0,
      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      1,255,252,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,0,0,255,255,248,0,0,0,0,0,0,0,0,
      0,127,255,255,0,0,0,0,0,0,0,0,0,0,0,
      0,1,127,255,0,0,0,0,0,0,0,0,0,0,0,0,
      3,254,0>>}}}}
      in function compaction_daemon:do_spawn_view_index_compactor/5
      in call from compaction_daemon:'spawn_view_index_compactor/6-fun-0'/7
      ancestors: [<0.2809.108>,compaction_daemon,<0.26296.30>,ns_server_sup,
      ns_server_cluster_sup,<0.67.0>]
      messages: []
      links: [<0.2809.108>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 377
      stack_size: 24
      reductions: 1694
      neighbours:

      [ns_server:info,2012-11-28T17:40:16.130,ns_1@10.224.47.35:<0.5017.108>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default`
      [error_logger:error,2012-11-28T17:40:16.130,ns_1@10.224.47.35:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: compaction_daemon:spawn_view_compactor/5-fun-1/0
      pid: <0.2809.108>
      registered_name: []
      exception exit: {updater_died,
      {updater_error,
      {badmatch,
      {ok,<<0,0,58,58,118,0,0,0,0,0,0,0,0,0,0,0,
      0,12,0,240,0,0,0,0,0,0,0,0,0,0,0,3,0,
      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      1,255,252,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,0,0,255,255,248,0,0,0,0,0,0,0,0,
      0,127,255,255,0,0,0,0,0,0,0,0,0,0,0,
      0,1,127,255,0,0,0,0,0,0,0,0,0,0,0,0,
      3,254,0>>}}}}
      in function compaction_daemon:do_chain_compactors/2
      ancestors: [compaction_daemon,<0.26296.30>,ns_server_sup,
      ns_server_cluster_sup,<0.67.0>]
      messages: []
      links: [<0.26297.30>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 4181
      stack_size: 24
      reductions: 4711
      neighbours:

      [error_logger:error,2012-11-28T17:40:16.130,ns_1@10.224.47.35:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: compaction_daemon:spawn_view_compactor/5-fun-1/0
      pid: <0.28230.107>
      registered_name: []
      exception exit: {updater_died,
      {updater_error,
      {badmatch,
      {ok,<<0,0,58,58,118,0,0,0,0,0,0,0,0,0,0,0,
      0,12,0,240,0,0,0,0,0,0,0,0,0,0,0,3,0,
      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      252,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
      1,255,252,0,0,0,0,0,0,0,0,0,0,0,0,0,
      0,0,0,0,255,255,248,0,0,0,0,0,0,0,0,
      0,127,255,255,0,0,0,0,0,0,0,0,0,0,0,
      0,1,127,255,0,0,0,0,0,0,0,0,0,0,0,0,
      3,254,0>>}}}}
      in function compaction_daemon:do_chain_compactors/2
      ancestors: [<0.13946.106>,<0.13945.106>,compaction_daemon,<0.26296.30>,
      ns_server_sup,ns_server_cluster_sup,<0.67.0>]
      messages: []
      links: [<0.13946.106>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 1597
      stack_size: 24
      reductions: 4646
      neighbours:

      Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1965-rel.setup.exe.manifest.xml

      # Subject Project Status CR V
      For Gerrit Dashboard: &For+MB-7285=message:MB-7285

        Activity

        Show
        thuan Thuan Nguyen added a comment - Link to diags from all nodes https://s3.amazonaws.com/bugdb/jira/MB-7285/8nodes-1965-manual-compaction-failed-20121128-141532.tgz
        Show
        thuan Thuan Nguyen added a comment - Link to all logs from all nodes https://s3.amazonaws.com/bugdb/jira/MB-7285/8nodes-1965-manual-compaction-failed-20121128.tgz
        Show
        FilipeManana Filipe Manana (Inactive) added a comment - http://review.couchbase.org/#/c/22947/
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #539 (See http://qa.hq.northscale.net/job/github-couchdb-preview/539/)
        MB-7285 Fix badmatch in index updater on cleanup (Revision 294b4aca9b249da86f1e3a621dff3aa757cde963)

        Result = SUCCESS
        Farshid Ghods :
        Files :

        • src/couch_set_view/src/couch_set_view_updater.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #539 (See http://qa.hq.northscale.net/job/github-couchdb-preview/539/ ) MB-7285 Fix badmatch in index updater on cleanup (Revision 294b4aca9b249da86f1e3a621dff3aa757cde963) Result = SUCCESS Farshid Ghods : Files : src/couch_set_view/src/couch_set_view_updater.erl
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Merged to 2.0.0 and master.
        However keeping this open as I'm unable to merge to 2.0.1 branch.
        Needs Farshid's help.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Merged to 2.0.0 and master. However keeping this open as I'm unable to merge to 2.0.1 branch. Needs Farshid's help.

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes