Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5549

rebalance failed because memcached drops the connection ( UI says "change_filter_failed" )

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0-beta
    • 1.8.1
    • couchbase-bucket
    • Security Level: Public
    • None
    • windows 2008 R2 64 bit

    Description

      Run regression test rebalancetests.RebalanceInOutWithParallelLoad.test_load,replica=2,do-stop=True
      During this test, rebalance failed with error:

      [ns_server:debug] [2012-06-13 8:22:26] [ns_1@10.1.3.82:<0.31153.111>:ns_replicas_builder:spawn_replica_builder:246] Replica building ebucketmigrator for vbucket 502 into 'ns_1@10.1.3.85' is <16961.29616.25>
      [ns_server:debug] [2012-06-13 8:22:26] [ns_1@10.1.3.82:<0.31153.111>:ns_replicas_builder:spawn_replica_builder:246] Replica building ebucketmigrator for vbucket 502 into 'ns_1@10.1.3.87' is <16962.25655.16>
      [ns_server:debug] [2012-06-13 8:22:26] [ns_1@10.1.3.82:<0.29704.111>:ns_vbm_sup:apply_changes:81] Applying changes:
      [

      {add_replica,'ns_1@10.1.3.87','ns_1@10.1.3.84',69}

      ,

      {add_replica,'ns_1@10.1.3.84','ns_1@10.1.3.82',69}

      ]

      [ns_server:debug] [2012-06-13 8:22:26] [ns_1@10.1.3.82:<0.31118.111>:ebucketmigrator_srv:init:228] killing tap named: replication_building_241_'ns_1@10.1.3.82'
      [rebalance:debug] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.29704.111>:ns_vbucket_mover:terminate:169] Savings: 0 (from 165)

      [ns_server:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31113.111>:ns_single_vbucket_mover:mover_inner:88] Got exit message (parent is <0.29704.111>). Exiting...
      {'EXIT',<0.29704.111>,
      {{change_filter_failed,
      {'EXIT',
      {normal,
      {gen_server,call,
      [<16690.2164.10>,start_vbucket_filter_change,30000]}}}},
      [

      {ns_vbm_sup,change_vbucket_filter,4},
      {ns_vbm_sup,'-set_replicas/3-fun-2-',5},
      {lists,foldl,3},
      {ns_vbm_sup,set_replicas,3},
      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',3},
      {lists,foreach,2},
      {ns_vbm_sup,apply_changes,2},
      {ns_vbucket_mover,sync_replicas,0}]}}
      [ns_server:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31075.111>:ns_single_vbucket_mover:mover_inner:88] Got exit message (parent is <0.29704.111>). Exiting...
      {'EXIT',<0.29704.111>,
      {{change_filter_failed,
      {'EXIT',
      {normal,
      {gen_server,call,
      [<16690.2164.10>,start_vbucket_filter_change,30000]}}}},
      [{ns_vbm_sup,change_vbucket_filter,4}

      ,

      {ns_vbm_sup,'-set_replicas/3-fun-2-',5},
      {lists,foldl,3},
      {ns_vbm_sup,set_replicas,3},
      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',3},
      {lists,foreach,2},
      {ns_vbm_sup,apply_changes,2},
      {ns_vbucket_mover,sync_replicas,0}]}}
      [ns_server:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31152.111>:ns_single_vbucket_mover:mover_inner:88] Got exit message (parent is <0.29704.111>). Exiting...
      {'EXIT',<0.29704.111>,
      {{change_filter_failed,
      {'EXIT',
      {normal,
      {gen_server,call,
      [<16690.2164.10>,start_vbucket_filter_change,30000]}}}},
      [{ns_vbm_sup,change_vbucket_filter,4},
      {ns_vbm_sup,'-set_replicas/3-fun-2-',5}

      ,

      {lists,foldl,3},
      {ns_vbm_sup,set_replicas,3},
      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',3},
      {lists,foreach,2},
      {ns_vbm_sup,apply_changes,2},
      {ns_vbucket_mover,sync_replicas,0}]}}
      [ns_server:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31114.111>:ns_replicas_builder:build_replicas_main:113] Got exit not from child ebucketgrator. Assuming it's our parent: {'EXIT', <0.31113.111>, shutdown}
      [ns_server:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31114.111>:ns_replicas_builder:kill_a_bunch_of_tap_names:219] Killed the following tap names on 'ns_1@10.1.3.84': [<<"replication_building_241_'ns_1@10.1.3.82'">>,
      <<"replication_building_241_'ns_1@10.1.3.87'">>]
      [error_logger:error] [2012-06-13 8:22:27] [ns_1@10.1.3.82:error_logger:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_single_vbucket_mover:mover/6
      pid: <0.31113.111>
      registered_name: []
      exception exit: {exited,
      {'EXIT',<0.29704.111>,
      {{change_filter_failed,
      {'EXIT',
      {normal,
      {gen_server,call,
      [<16690.2164.10>,start_vbucket_filter_change,
      30000]}}}},
      [{ns_vbm_sup,change_vbucket_filter,4},
      {ns_vbm_sup,'-set_replicas/3-fun-2-',5},
      {lists,foldl,3}

      ,

      {ns_vbm_sup,set_replicas,3},
      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',3},
      {lists,foreach,2},
      {ns_vbm_sup,apply_changes,2},
      {ns_vbucket_mover,sync_replicas,0}]}}}
      in function ns_single_vbucket_mover:mover_inner/6
      in call from ns_replicas_builder:try_with_maybe_ignorant_after/2
      in call from ns_single_vbucket_mover:mover/6
      ancestors: [<0.29704.111>,<0.29682.111>]
      messages: [{'EXIT',<0.29704.111>,
      {{change_filter_failed,
      {'EXIT',
      {normal,
      {gen_server,call,
      [<16690.2164.10>,
      start_vbucket_filter_change,30000]}}}},
      [{ns_vbm_sup,change_vbucket_filter,4},
      {ns_vbm_sup,'-set_replicas/3-fun-2-',5},
      {lists,foldl,3},
      {ns_vbm_sup,set_replicas,3}

      ,

      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',3}

      ,

      {lists,foreach,2}

      ,

      {ns_vbm_sup,apply_changes,2}

      ,

      {ns_vbucket_mover,sync_replicas,0}

      ]}}]
      links: [<0.29704.111>]
      dictionary: [

      {cleanup_list,[<0.31114.111>]}

      ]
      trap_exit: true
      status: running
      heap_size: 2584
      stack_size: 24
      reductions: 2909
      neighbours:

      [rebalance:info] [2012-06-13 8:22:27] [ns_1@10.1.3.82:<0.31151.111>:ebucketmigrator_srv:init:252] Starting tap stream:
      [{vbuckets,[86,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,
      357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,

      Log file store here:
      https://s3.amazonaws.com/packages.couchbase/diag-logs/log-rebalance-change_filter_failed.tgz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty