Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11325

{UPR}:: Rebalance hanging during View Query + Rebalance tests

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0
    • 3.0
    • Security Level: Public
    • None
    • 3.0.0-780-rel
    • Triaged
    • Ubuntu 64-bit
    • recent logs (10.6.2.144-146 to be considered)
    • Yes
    • June 30 - July 18

    Description

      Jenkins Ref Link:
      http://qa.sc.couchbase.com/job/ubuntu_x64--65_02--view_query_extended-P1/94/consoleFull

      Test to Reproduce:
      ./testrunner -i <yourfile>.ini get-delays=True,get-cbcollect-info=True,GROUP=P1 -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,limit=1,skip_rebalance=true,GROUP=P1

      2014-06-05 04:34:34 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.106.196%2Cns_1%40172.23.106.197
      2014-06-05 04:38:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 46.875 %
      ....
      2014-06-05 04:50:31 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 46.875 %

      Logs:

      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_memcached:init/1
      pid: <0.11634.1>
      registered_name: []
      exception exit: {badmatch,{error,closed}}
      in function gen_server:init_it/6 (gen_server.erl, line 328)
      ancestors: ['single_bucket_sup-default',<0.11617.1>]
      messages: []
      links: [<0.11660.1>,<0.11662.1>,<0.11663.1>,<0.277.0>,<0.11618.1>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 75113
      stack_size: 27
      reductions: 2856901
      neighbours:
      neighbour: [

      {pid,<0.11663.1>}

      ,

      {registered_name,[]},
      {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
      {current_function,{gen,do_call,4}},
      {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.11617.1>]},
      {messages,[]},
      {links,[<0.11634.1>,#Port<0.11442>]},
      {dictionary,[]},
      {trap_exit,false},
      {status,waiting},
      {heap_size,46422},
      {stack_size,23},
      {reductions,272768723}]
      neighbour: [{pid,<0.11662.1>},
      {registered_name,[]}

      ,
      {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
      {current_function,{gen,do_call,4}},

      {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.11617.1>]},
      {messages,[]},
      {links,[<0.11634.1>,#Port<0.11443>]},
      {dictionary,[]},
      {trap_exit,false},
      {status,waiting},
      {heap_size,28690},
      {stack_size,23},
      {reductions,2905}]
      neighbour: [{pid,<0.11660.1>},
      {registered_name,[]},
      {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
      {current_function,{gen,do_call,4}},
      {ancestors,['ns_memcached-default', 'single_bucket_sup-default',<0.11617.1>]}

      ,

      {messages,[]}

      ,

      {links,[<0.11634.1>,#Port<0.11440>]}

      ,

      {dictionary,[]}

      ,

      {trap_exit,false}

      ,

      {status,waiting}

      ,

      {heap_size,46422}

      ,

      {stack_size,23}

      ,

      {reductions,258119319}

      ]

      [error_logger:error,2014-06-05T4:52:22.360,ns_1@172.23.106.196:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]Supervisor received unexpected message: {ack,<0.11634.1>,
      {error,{badmatch,{error,closed}}}}

      [ns_server:info,2014-06-05T4:52:22.498,ns_1@172.23.106.196:janitor_agent-default<0.11640.1>:janitor_agent:handle_info:817]Undoing temporary vbucket states caused by rebalance
      [user:info,2014-06-05T4:52:22.499,ns_1@172.23.106.196:<0.1173.0>:ns_orchestrator:handle_info:480]Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.23026.2>,
      {{noproc,
      {gen_server,call,
      ['replication_manager-default',

      {upr_takeover,'ns_1@172.23.106.196',641},
      infinity]}},
      {gen_server,call,
      [{'janitor_agent-default', 'ns_1@172.23.106.197'},
      {if_rebalance,<0.6656.2>,
      {wait_index_updated,655}},
      infinity]}}}}

      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_single_vbucket_mover:upr_takeover/5-fun-0/0
      pid: <0.29768.2>
      registered_name: []
      exception exit: {{noproc,
      {gen_server,call,
      ['replication_manager-default',
      {upr_takeover,'ns_1@172.23.106.196',641}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-default','ns_1@172.23.106.197'}

      ,
      {if_rebalance,<0.6656.2>,
      {upr_takeover,'ns_1@172.23.106.196',641}},
      infinity]}}
      in function gen_server:call/3 (gen_server.erl, line 188)
      in call from ns_single_vbucket_mover:'upr_takeover/5-fun-0'/5 (src/ns_single_vbucket_mover.erl, line 307)
      ancestors: [<0.23372.2>,<0.6656.2>,<0.6610.2>,<0.1173.0>,mb_master_sup,
      mb_master,ns_server_sup,ns_server_cluster_sup,<0.59.0>]
      messages: []
      links: [<0.23372.2>]
      dictionary: []
      trap_exit: false
      status: running
      heap_size: 610
      stack_size: 27
      reductions: 125
      neighbours:

      Uploading Logs.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mikew Mike Wiederhold [X] (Inactive)
              Meenakshi Meenakshi Goel
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty