Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43928

[REGRESSION] cannot online upgrade analytics to Cheshire-Cat

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • ns_server
    • Untriaged
    • 1
    • Yes

    Description

      Online upgrade scenarios are failing, due to upgraded analytics nodes never becoming registered in cbauth in cbas.
      e.g. from UpgradeFromAlice600FailoverITD_failoverUpgradeAnalytics.zip 0004_cbas2_thisbuild/logs/analytics_cbas_debug.log

      2021-01-28T04:11:07.254+00:00 DEBU CBAS.cbas cbauth.GetMemcachedServiceAuth(cbas2.couchbase.host:8095) returned error: Unable to find given hostport in cbauth database: `cbas2.couchbase.host:8095'
      

      this is working at c2ee65932e1c552a76553e7f8e16f5f34b54ce6e : ns_server, so is a regression since then.

      This is not intermittent- all online upgrade scenarios fail consistently.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-4334 contains cbas-core commit 30dbd2b with commit message:
            MB-43928: disable regressed test cases

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4334 contains cbas-core commit 30dbd2b with commit message: MB-43928 : disable regressed test cases
            dfinlay Dave Finlay added a comment -

            Thanks Mike.

            Artem Stemkovski: Looks like this might be a fairly straightforward bug - though it's entirely possible I've misanalyzed it: I don't see the keys of the form:

            {service_maps, _}
            

            get migrated to chronicle.

            Here's the report on the keys that are upgraded:

            [ns_server:info,2021-01-28T04:08:07.528Z,ns_1@cbas1.couchbase.host:ns_config<0.214.0>:chronicle_local:do_upgrade:252]Keys are migrated to chronicle. Rev = {<<"71095af26a8a396f386eb06691bc0ebd">>,
                                                   7}. Sets = [{set,bucket_names,
                                                                ["beer-sample"]},
                                                               {set,nodes_wanted,
                                                                ['ns_1@cbas1.couchbase.host',
                                                                 'ns_1@cbas2.couchbase.host',
                                                                 'ns_1@kv1new.couchbase.host']},
                                                               {set,server_groups,
                                                                [[{uuid,<<"0">>},
                                                                  {name,<<"Group 1">>},
                                                                  {nodes,
                                                                   ['ns_1@cbas1.couchbase.host',
                                                                    'ns_1@cbas2.couchbase.host',
                                                                    'ns_1@kv1new.couchbase.host']}]]},
                                                               {set,
                                                                {bucket,"beer-sample",
                                                                 collections},
                                                                [{uid,0},
                                                                 {next_scope_uid,7},
                                                                 {next_coll_uid,7},
                                                                 {num_scopes,0},
                                                                 {num_collections,0},
                                                                 {scopes,
                                                                  [{"_default",
                                                                    [{uid,0},
                                                                     {collections,
                                                                      [{"_default",
                                                                        [{uid,0}]}]}]}]}]},
                                                               {set,
                                                                {node,'ns_1@127.0.0.1',
                                                                 membership},
                                                                active},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas1.couchbase.host',
                                                                 failover_vbuckets},
                                                                []},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas1.couchbase.host',
                                                                 membership},
                                                                active},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas1.couchbase.host',
                                                                 recovery_type},
                                                                none},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas1.couchbase.host',
                                                                 services},
                                                                [cbas]},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas2.couchbase.host',
                                                                 failover_vbuckets},
                                                                []},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas2.couchbase.host',
                                                                 membership},
                                                                active},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas2.couchbase.host',
                                                                 recovery_type},
                                                                none},
                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas2.couchbase.host',
                                                                 services},
                                                                [cbas]},
                                                               {set,
                                                                {node,
                                                                 'ns_1@kv1new.couchbase.host',
                                                                 failover_vbuckets},
                                                                []},
                                                               {set,
                                                                {node,
                                                                 'ns_1@kv1new.couchbase.host',
                                                                 membership},
                                                                active},
                                                               {set,
                                                                {node,
                                                                 'ns_1@kv1new.couchbase.host',
                                                                 recovery_type},
                                                                none},
                                                               {set,
                                                                {node,
                                                                 'ns_1@kv1new.couchbase.host',
                                                                 services},
                                                                [kv]}]
            

            Node services get migrated right:

                                                               {set,
                                                                {node,
                                                                 'ns_1@cbas1.couchbase.host',
                                                                 services},
                                                                [cbas]},
            

            But not the per-service service_maps.

            dfinlay Dave Finlay added a comment - Thanks Mike. Artem Stemkovski : Looks like this might be a fairly straightforward bug - though it's entirely possible I've misanalyzed it: I don't see the keys of the form: {service_maps, _} get migrated to chronicle. Here's the report on the keys that are upgraded: [ns_server:info,2021-01-28T04:08:07.528Z,ns_1@cbas1.couchbase.host:ns_config<0.214.0>:chronicle_local:do_upgrade:252]Keys are migrated to chronicle. Rev = {<<"71095af26a8a396f386eb06691bc0ebd">>, 7}. Sets = [{set,bucket_names, ["beer-sample"]}, {set,nodes_wanted, ['ns_1@cbas1.couchbase.host', 'ns_1@cbas2.couchbase.host', 'ns_1@kv1new.couchbase.host']}, {set,server_groups, [[{uuid,<<"0">>}, {name,<<"Group 1">>}, {nodes, ['ns_1@cbas1.couchbase.host', 'ns_1@cbas2.couchbase.host', 'ns_1@kv1new.couchbase.host']}]]}, {set, {bucket,"beer-sample", collections}, [{uid,0}, {next_scope_uid,7}, {next_coll_uid,7}, {num_scopes,0}, {num_collections,0}, {scopes, [{"_default", [{uid,0}, {collections, [{"_default", [{uid,0}]}]}]}]}]}, {set, {node,'ns_1@127.0.0.1', membership}, active}, {set, {node, 'ns_1@cbas1.couchbase.host', failover_vbuckets}, []}, {set, {node, 'ns_1@cbas1.couchbase.host', membership}, active}, {set, {node, 'ns_1@cbas1.couchbase.host', recovery_type}, none}, {set, {node, 'ns_1@cbas1.couchbase.host', services}, [cbas]}, {set, {node, 'ns_1@cbas2.couchbase.host', failover_vbuckets}, []}, {set, {node, 'ns_1@cbas2.couchbase.host', membership}, active}, {set, {node, 'ns_1@cbas2.couchbase.host', recovery_type}, none}, {set, {node, 'ns_1@cbas2.couchbase.host', services}, [cbas]}, {set, {node, 'ns_1@kv1new.couchbase.host', failover_vbuckets}, []}, {set, {node, 'ns_1@kv1new.couchbase.host', membership}, active}, {set, {node, 'ns_1@kv1new.couchbase.host', recovery_type}, none}, {set, {node, 'ns_1@kv1new.couchbase.host', services}, [kv]}] Node services get migrated right: {set, {node, 'ns_1@cbas1.couchbase.host', services}, [cbas]}, But not the per-service service_maps.

            Build couchbase-server-7.0.0-4342 contains ns_server commit 06940ad with commit message:
            MB-43928 correctly upgrade service related keys to chronicle

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4342 contains ns_server commit 06940ad with commit message: MB-43928 correctly upgrade service related keys to chronicle

            Build couchbase-server-7.0.0-4366 contains cbas-core commit 5272a09 with commit message:
            Revert "MB-43928: disable regressed test cases"

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4366 contains cbas-core commit 5272a09 with commit message: Revert " MB-43928 : disable regressed test cases"
            umang.agrawal Umang added a comment -

            Verified with build 7.0.0-4825

            umang.agrawal Umang added a comment - Verified with build 7.0.0-4825

            People

              umang.agrawal Umang
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty