Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25677

Graceful failover fails during upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • 4.6.2, 5.0.0
    • 4.6.2, 5.0.0
    • couchbase-bucket
    • None
    • Enterprise Edition 4.6.2 build 3905
      Enterprise Edition 5.0.0 build 3491

    Description

      Steps to Repro.
      1) created a 5 node cluster 2 kv, 2 index and 1 n1ql in 4.6.2-3905
      2) Created 3 buckets with each a 2 million docs in each bucket.
      3) Created replica indexes.
      4) failedover one of the indexer and upgraded(to Enterprise Edition 5.0.0 build 3491) followed by full recovery with concurrent querying and kv ops.
      5) failedover one of the kv node and upgraded(to Enterprise Edition 5.0.0 build 3491) followed by full recovery with concurrent querying and kv ops.
      6) failedover other indexer and upgraded(to Enterprise Edition 5.0.0 build 3491) followed by full recovery with concurrent querying and kv ops.
      7) Tried to failover on the last remaining the kv node for upgrade with concurrent querying and kv ops.

      Graceful failover fails. Tried this couple of times. Failed both times with following error.

      Graceful failover exited with reason {unexpected_exit,
                                            {'EXIT',<0.9080.23>,
                                             {wait_seqno_persisted_failed,
                                              "bucket2",983,76663,
                                              [{'ns_1@172.23.122.13',
                                                {'EXIT',
                                                 {{{child_interrupted,
                                                    {'EXIT',<12730.842.7>,
                                                     socket_closed}},
                                                   [{dcp_replicator,spawn_and_wait,
                                                     1,
                                                     [{file,
                                                       "src/dcp_replicator.erl"},
                                                      {line,231}]},
                                                    {dcp_replicator,handle_call,3,
                                                     [{file,
                                                       "src/dcp_replicator.erl"},
                                                      {line,115}]},
                                                    {gen_server,handle_msg,5,
                                                     [{file,"gen_server.erl"},
                                                      {line,585}]},
                                                    {proc_lib,init_p_do_apply,3,
                                                     [{file,"proc_lib.erl"},
                                                      {line,239}]}]},
                                                  {gen_server,call,
                                                   [{'janitor_agent-bucket2',
                                                     'ns_1@172.23.122.13'},
                                                    {if_rebalance,<0.5403.23>,
                                                     {wait_seqno_persisted,983,
                                                      76663}},
                                                    infinity]}}}}]}}}
      

      <0.6347.23> exited with {unexpected_exit,
                               {'EXIT',<0.9080.23>,
                                {wait_seqno_persisted_failed,"bucket2",983,76663,
                                 [{'ns_1@172.23.122.13',
                                   {'EXIT',
                                    {{{child_interrupted,
                                       {'EXIT',<12730.842.7>,socket_closed}},
                                      [{dcp_replicator,spawn_and_wait,1,
                                        [{file,"src/dcp_replicator.erl"},
                                         {line,231}]},
                                       {dcp_replicator,handle_call,3,
                                        [{file,"src/dcp_replicator.erl"},
                                         {line,115}]},
                                       {gen_server,handle_msg,5,
                                        [{file,"gen_server.erl"},{line,585}]},
                                       {proc_lib,init_p_do_apply,3,
                                        [{file,"proc_lib.erl"},{line,239}]}]},
                                     {gen_server,call,
                                      [{'janitor_agent-bucket2',
                                        'ns_1@172.23.122.13'},
                                       {if_rebalance,<0.5403.23>,
                                        {wait_seqno_persisted,983,76663}},
                                       infinity]}}}}]}}}
      

      Host Config:

      172.23.107.185 - n1ql
      172.23.122.13 - kv
      172.23.122.58 - kv
      172.23.122.59 - index
      172.23.122.60 - index
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty