Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45906

[CE] : Rebalance fails when adding a node to the cluster - badmatch,failed

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5017
      Last known good build : 7.0.0-4960

      Steps :
      1. Create a 1 node cluster with CE build with all services - KV, index, query, search
      2. Add one more node to the cluster with all services - KV, index, query, search
      3. Rebalance.

      Rebalance fails with error -
      Rebalance exited with reason {{badmatch,failed},
      [{ns_rebalancer,rebalance_body,5,
      [

      {file,"src/ns_rebalancer.erl"}

      ,

      {line,543}

      ]},
      {async,'async_init/4-fun-1',3,
      [

      {file,"src/async.erl"}

      ,

      {line,191}

      ]}]}.
      Rebalance Operation Id = 93e388d6f43d5408b0bd08c5ff17f092

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          dfinlay Dave Finlay added a comment -

          This is a regression caused by this change: http://review.couchbase.org/c/ns_server/+/151560.

          SSL REST ports are not available in CE configs.

          Abhinav Dangeti: ns_server will change this to only include the SSL REST port on the command line in EE versions. Can you make sure the code of the FTS side can handle it in CE?

          dfinlay Dave Finlay added a comment - This is a regression caused by this change: http://review.couchbase.org/c/ns_server/+/151560 . SSL REST ports are not available in CE configs. Abhinav Dangeti : ns_server will change this to only include the SSL REST port on the command line in EE versions. Can you make sure the code of the FTS side can handle it in CE?
          dfinlay Dave Finlay added a comment -

          Here's the backtrace that indicates it's the ssl_rest_port:

          [ns_server:debug,2021-04-25T18:55:50.756-07:00,ns_1@127.0.0.1:<0.4316.0>:remote_monitors:handle_down:151]Caller of remote monitor <0.569.0> died with {{badmatch,{ssl_rest_port,false}},
                                                        [{ns_ports_setup,
                                                          build_port_arg,4,
                                                          [{file,
                                                            "src/ns_ports_setup.erl"},
                                                           {line,266}]},
                                                         {ns_ports_setup,
                                                          '-build_port_args/2-lc$^0/1-0-',
                                                          2,
                                                          [{file,
                                                            "src/ns_ports_setup.erl"},
                                                           {line,270}]},
                                                         {ns_ports_setup,goport_args,4,
                                                          [{file,
                                                            "src/ns_ports_setup.erl"},
                                                           {line,507}]},
                                                         {ns_ports_setup,
                                                          build_goport_spec,3,
                                                          [{file,
                                                            "src/ns_ports_setup.erl"},
                                                           {line,341}]},
                                                         {ns_ports_setup,
                                                          '-build_goport_specs/2-lc$^0/1-0-',
                                                          3,
                                                          [{file,
                                                            "src/ns_ports_setup.erl"},
                                                           {line,348}]},
          

          dfinlay Dave Finlay added a comment - Here's the backtrace that indicates it's the ssl_rest_port: [ns_server:debug,2021-04-25T18:55:50.756-07:00,ns_1@127.0.0.1:<0.4316.0>:remote_monitors:handle_down:151]Caller of remote monitor <0.569.0> died with {{badmatch,{ssl_rest_port,false}}, [{ns_ports_setup, build_port_arg,4, [{file, "src/ns_ports_setup.erl"}, {line,266}]}, {ns_ports_setup, '-build_port_args/2-lc$^0/1-0-', 2, [{file, "src/ns_ports_setup.erl"}, {line,270}]}, {ns_ports_setup,goport_args,4, [{file, "src/ns_ports_setup.erl"}, {line,507}]}, {ns_ports_setup, build_goport_spec,3, [{file, "src/ns_ports_setup.erl"}, {line,341}]}, {ns_ports_setup, '-build_goport_specs/2-lc$^0/1-0-', 3, [{file, "src/ns_ports_setup.erl"}, {line,348}]},

          Thanks Dave Finlay. FTS already just falls back to using the non-SSL port if ns_server doesn't hand it the SSL port, so we should be set there.

          abhinav Abhinav Dangeti added a comment - Thanks Dave Finlay . FTS already just falls back to using the non-SSL port if ns_server doesn't hand it the SSL port, so we should be set there.

          Ok, so I will modify this to conditionally include it based on whether we are CE or EE. 

          bryan.mccoid Bryan McCoid added a comment - Ok, so I will modify this to conditionally include it based on whether we are CE or EE. 

          Build couchbase-server-7.0.0-5041 contains ns_server commit 86d4353 with commit message:
          MB-45906: Fix goport args discrepancy between CE/EE versions

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5041 contains ns_server commit 86d4353 with commit message: MB-45906 : Fix goport args discrepancy between CE/EE versions

          Verified on 7.0.0-5071

          mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified on 7.0.0-5071

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty