Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35438

[System Test]: Rebalance failed due to queuedSeqnoAck invariant failed

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-3939

      Test: longevity 

      Cycle: 3rd

      Test step:

      [2019-08-03T03:30:02-07:00, sequoiatools/couchbase-cli:6.5:952ced] rebalance -c 172.23.108.103:8091 --server-remove 172.23.106.100:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.106.100:8091 -u Administrator -p password]
       
      docker logs 952ced
      docker start 952ced
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-08-03T03:34:28-07:00, sequoiatools/cmd:a01523] 60 

      Error:

      [ns_server:error,2019-08-03T03:34:04.376-07:00,ns_1@172.23.108.103:<0.21031.811>:ns_single_vbucket_mover:spawn_and_wait:82]Got unexpected exit signal {'EXIT',<0.30127.811>, {unexpected_exit, {'EXIT',<0.29171.811>, {{bulk_set_vbucket_state_failed, [{'ns_1@172.23.97.239', {'EXIT', {{{{{child_interrupted, {'EXIT',<25656.19748.455>, socket_closed}}, [{dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,249}]}, {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]}, {gen_server,try_handle_call,4, [{file,"gen_server.erl"},{line,636}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"},{line,665}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,247}]}]}, {gen_server,call, [<25656.19789.455>,get_partitions, infinity]}}, {gen_server,call, ['dcp_replication_manager-DISTRICT', {get_replicator_pid,633}, infinity]}}, {gen_server,call, [{'janitor_agent-DISTRICT', 'ns_1@172.23.97.239'}, {if_rebalance,<0.30127.811>, {update_vbucket_state,363,replica, undefined,undefined}}, infinity]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4, [{file,"src/janitor_agent.erl"},{line,432}]}, {ns_single_vbucket_mover, '-cleanup_old_streams/4-fun-1-',4, [{file,"src/ns_single_vbucket_mover.erl"}, {line,353}]}, {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}
       
      [user:error,2019-08-03T03:34:04.384-07:00,ns_1@172.23.108.103:<0.20612.277>:ns_orchestrator:log_rebalance_completion:1474]Rebalance exited with reason {mover_crashed, {unexpected_exit, {'EXIT',<0.29171.811>, {{bulk_set_vbucket_state_failed, [{'ns_1@172.23.97.239', {'EXIT', {{{{{child_interrupted, {'EXIT',<25656.19748.455>, socket_closed}}, [{dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,249}]}, {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,121}]}, {gen_server,try_handle_call,4, [{file,"gen_server.erl"}, {line,636}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"}, {line,665}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,247}]}]}, {gen_server,call, [<25656.19789.455>,get_partitions, infinity]}}, {gen_server,call, ['dcp_replication_manager-DISTRICT', {get_replicator_pid,633}, infinity]}}, {gen_server,call, [{'janitor_agent-DISTRICT', 'ns_1@172.23.97.239'}, {if_rebalance,<0.30127.811>, {update_vbucket_state,363,replica, undefined,undefined}}, infinity]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4, [{file,"src/janitor_agent.erl"}, {line,432}]},{ns_single_vbucket_mover, '-cleanup_old_streams/4-fun-1-',4, [{file,"src/ns_single_vbucket_mover.erl"}, {line,353}]}, {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,232}]}]}}}}. Rebalance Operation Id = ebdb8b40ac8b6b538ce76836476bfac1 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty