Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19687

[system tests windows] Rebalance in exited with reason wait_seqno_persisted_failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.1
    • 4.5.0
    • couchbase-bucket
    • 4.5.0-2566
    • Untriaged
    • Unknown

    Description

      rebalance in 3+1 node on src cluster afte dataload more then 1 day

       
      Rebalance exited with reason {unexpected_exit,
                                    {'EXIT',<0.11916.96>,
                                     {wait_seqno_persisted_failed,"UserInfo",445,
                                      329,
                                      [{'ns_1@172.23.105.87',
                                        {'EXIT',
                                         {badarg,
                                          {gen_server,call,
                                           [{'janitor_agent-UserInfo',
                                             'ns_1@172.23.105.87'},
                                            {if_rebalance,<0.5644.95>,
                                             {wait_seqno_persisted,445,329}},
                                            infinity]}}}}]}}}
      ns_orchestrator 002	ns_1@172.23.105.87	11:58:02 PM Thu May 19, 2016
      <0.21947.95> exited with {unexpected_exit,
                                {'EXIT',<0.11916.96>,
                                 {wait_seqno_persisted_failed,"UserInfo",445,329,
                                  [{'ns_1@172.23.105.87',
                                    {'EXIT',
                                     {badarg,
                                      {gen_server,call,
                                       [{'janitor_agent-UserInfo',
                                         'ns_1@172.23.105.87'},
                                        {if_rebalance,<0.5644.95>,
                                         {wait_seqno_persisted,445,329}},
                                        infinity]}}}}]}}}	ns_vbucket_mover 000	ns_1@172.23.105.87	11:58:01 PM Thu May 19, 2016
      Bucket "AbRegNums" loaded on node 'ns_1@172.23.105.94' in 0 seconds.	ns_memcached 000	ns_1@172.23.105.94	11:48:59 PM Thu May 19, 2016
      Control connection to memcached on 'ns_1@172.23.105.94' disconnected: {case_clause,
                                                                             [<<"1342622819">>]}	ns_memcached 000	ns_1@172.23.105.94	11:48:36 PM Thu May 19, 2016
      Bucket "MsgsCalls" loaded on node 'ns_1@172.23.105.94' in 0 seconds.	ns_memcached 000	ns_1@172.23.105.94	11:48:22 PM Thu May 19, 2016
      Control connection to memcached on 'ns_1@172.23.105.94' disconnected: {badmatch,
                                                                             <<129,16,
                                                                               0,22,0,
                                                                               0,0,0,0,
                                                                               0,0,23,
                                                                               0,0,0,0,
                                                                               0,0,0,0,
                                                                               0,0,0,0,
                                                                               99,104,
                                                                               101,99,
                                                                               107,112,
                                                                               111,105,
                             ... show	ns_memcached 000	ns_1@172.23.105.94	11:48:22 PM Thu May 19, 2016
      Bucket "RevAB" loaded on node 'ns_1@172.23.105.94' in 0 seconds.	ns_memcached 000	ns_1@172.23.105.94	11:48:19 PM Thu May 19, 2016
      Control connection to memcached on 'ns_1@172.23.105.94' disconnected: {case_clause,
                                                                             [<<"replica">>]}	ns_memcached 000	ns_1@172.23.105.94	11:48:18 PM Thu May 19, 2016
      Bucket "UserInfo" rebalance does not seem to be swap rebalance	ns_vbucket_mover 000	ns_1@172.23.105.87	11:34:38 PM Thu May 19, 2016
      Bucket "UserInfo" loaded on node 'ns_1@172.23.107.48' in 0 seconds.	ns_memcached 000	ns_1@172.23.107.48	11:34:34 PM Thu May 19, 2016
      Started rebalancing bucket UserInfo	ns_rebalancer 000	ns_1@172.23.105.87	11:34:33 PM Thu May 19, 2016
      Starting rebalance, KeepNodes = ['ns_1@172.23.105.87','ns_1@172.23.105.94',
                                       'ns_1@172.23.107.48','ns_1@172.23.107.85'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004	ns_1@172.23.105.87	11:34:31 PM Thu May 19, 2016
      
      

      Attachments

        1. cpu-utilization.png
          cpu-utilization.png
          134 kB
        2. cpu-utilization-94.png
          cpu-utilization-94.png
          116 kB
        3. Diskdrive.txt
          2 kB
        4. dropped-ticks.png
          dropped-ticks.png
          206 kB
        5. dropped-ticks-94.png
          dropped-ticks-94.png
          270 kB
        6. logicaldisk.txt
          2 kB
        7. operations-94.png
          operations-94.png
          99 kB

        Issue Links

          For Gerrit Dashboard: MB-19687
          # Subject Branch Project Status CR V

          Activity

            People

              drigby Dave Rigby (Inactive)
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty