Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32447

Upgrade: Service 'projector' exited with status 2 in mix mode cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 6.0.1
    • 6.0.1
    • ns_server
    • None
    • Untriaged
    • Unknown

    Description

      Build: 6.0.1-2013

      Projector is crashing , after online upgrade from 5.5.0-2958 to 6.0.1-2013

      Service 'projector' exited with status 2. Restarting. Messages:
      2018/12/27 01:02:54 revrpc: Got error (EOF) and will retry in 1s
      2018-12-27T01:02:54.460-08:00 [Error] GetSettingsConfig() failed: Get http://127.0.0.1:8091/_metakv/indexing/settings/config: CBAuth database is stale: last reason: EOF
      panic: Get http://127.0.0.1:8091/_metakv/indexing/settings/config: CBAuth database is stale: last reason: EOF
       
      goroutine 1 [running]:
      panic(0xaba7e0, 0xc420110ae0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
      github.com/couchbase/indexing/secondary/common.CrashOnError(0x1259200, 0xc420110ae0)
      goproj/src/github.com/couchbase/indexing/secondary/common/util.go:406 +0x5c
      github.com/couchbase/indexing/secondary/projector.NewProjector(0x400, 0xc4202580f0, 0xc420163e88)
      goproj/src/github.com/couchbase/indexing/secondary/projector/projector.go:54 +0x177
      main.main()
      goproj/src/github.com/couchbase/indexing/secondary/cmd/projector/main.go:111 +0x70d
      [gopor ...showns_log 000ns_1@172.23.121.261:02:54 AM   Thu Dec 27, 2018
       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          The problem seems to be because of the following unmarshalling error in cbauth:

          "json: cannot unmarshal string into Go struct field Node.Ports of type int"

          Because of this, the cbauth database doesn't get populated at all and hence the clients get "CBAuth database is stale" error when they try to access cbauth. The following is the rev-rpc interaction that results in the unmarshalling error:

          [json_rpc:debug,2018-12-27T01:00:34.995-08:00,ns_1@172.23.121.17:json_rpc_connection-goxdcr-cbauth<0.23590.4>:json_rpc_connection:handle_call:158]sending jsonrpc call:{[{jsonrpc,<<"2.0">>},
           {id,0},
           {method,<<"AuthCacheSvc.UpdateDB">>},
           {params,
           [{[{specialUser,<<"@goxdcr-cbauth">>},
           {nodes,
           [{[{host,<<"172.23.98.196">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,
           [8091,18091,18092,8092,11207,9999,11210,
           11211]}]},
           {[{host,<<"172.23.121.26">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,
           [8091,18091,18092,8092,11207,9999,11210,
           11211]}]},
           {[{host,<<"172.23.121.17">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,
           [11210,8091,18091,9100,9101,9102,9103,9104,
           9105,19102]},
           {local,true}]},
           {[{host,<<"172.23.120.206">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,
           [8091,18091,9100,9101,9102,9103,9104,9105,
           19102]}]},
           {[{host,<<"172.23.120.201">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,[8091,18091,8096,undefined,18096]}]},
           {[{host,<<"172.23.120.192">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,[8091,18091]}]},
           {[{host,<<"172.23.120.177">>},
           {user,<<"@goxdcr">>},
           {password,"*****"},
           {ports,[8091,18091]}]}]},
           {authCheckURL,<<"http://127.0.0.1:8091/_cbauth">>},
           {permissionCheckURL,
           <<"http://127.0.0.1:8091/_cbauth/checkPermission">>},
           {permissionsVersion,
           <<"Qjdco6NCP7ur/RmYZCQr36o7atY=">>},
           {authVersion,<<"/xnnJ4ADQgDRAkZR2JzF6PMTdwQ=">>},
           {certVersion,3337661542},
           {extractUserFromCertURL,
           <<"http://127.0.0.1:8091/_cbauth/extractUserFromCert">>},
           {clientCertAuthState,<<"disable">>},
           {clientCertAuthVersion,
           <<"cAN8XSFAtQ98DdNbAKta8iJSRpw=">>}]}]}]}
          [json_rpc:debug,2018-12-27T01:00:34.996-08:00,ns_1@172.23.121.17:json_rpc_connection-goxdcr-cbauth<0.23590.4>:json_rpc_connection:handle_info:94]got response: [{<<"id">>,0},
           {<<"result">>,null},
           {<<"error">>,
           <<"json: cannot unmarshal string into Go struct field Node.Ports of type int">>}]

          Will put out a patch soon.

          ajit.yagaty Ajit Yagaty [X] (Inactive) added a comment - - edited The problem seems to be because of the following unmarshalling error in cbauth: "json: cannot unmarshal string into Go struct field Node.Ports of type int" Because of this, the cbauth database doesn't get populated at all and hence the clients get "CBAuth database is stale" error when they try to access cbauth. The following is the rev-rpc interaction that results in the unmarshalling error: [json_rpc:debug, 2018 - 12 -27T01: 00 : 34.995 - 08 : 00 ,ns_1 @172 .23. 121.17 :json_rpc_connection-goxdcr-cbauth< 0.23590 . 4 >:json_rpc_connection:handle_call: 158 ]sending jsonrpc call:{[{jsonrpc,<< "2.0" >>}, {id, 0 }, {method,<< "AuthCacheSvc.UpdateDB" >>}, {params, [{[{specialUser,<< "@goxdcr-cbauth" >>}, {nodes, [{[{host,<< "172.23.98.196" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports, [ 8091 , 18091 , 18092 , 8092 , 11207 , 9999 , 11210 , 11211 ]}]}, {[{host,<< "172.23.121.26" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports, [ 8091 , 18091 , 18092 , 8092 , 11207 , 9999 , 11210 , 11211 ]}]}, {[{host,<< "172.23.121.17" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports, [ 11210 , 8091 , 18091 , 9100 , 9101 , 9102 , 9103 , 9104 , 9105 , 19102 ]}, {local, true }]}, {[{host,<< "172.23.120.206" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports, [ 8091 , 18091 , 9100 , 9101 , 9102 , 9103 , 9104 , 9105 , 19102 ]}]}, {[{host,<< "172.23.120.201" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports,[ 8091 , 18091 , 8096 ,undefined, 18096 ]}]}, {[{host,<< "172.23.120.192" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports,[ 8091 , 18091 ]}]}, {[{host,<< "172.23.120.177" >>}, {user,<< "@goxdcr" >>}, {password, "*****" }, {ports,[ 8091 , 18091 ]}]}]}, {authCheckURL,<< "http://127.0.0.1:8091/_cbauth" >>}, {permissionCheckURL, << "http://127.0.0.1:8091/_cbauth/checkPermission" >>}, {permissionsVersion, << "Qjdco6NCP7ur/RmYZCQr36o7atY=" >>}, {authVersion,<< "/xnnJ4ADQgDRAkZR2JzF6PMTdwQ=" >>}, {certVersion, 3337661542 }, {extractUserFromCertURL, << "http://127.0.0.1:8091/_cbauth/extractUserFromCert" >>}, {clientCertAuthState,<< "disable" >>}, {clientCertAuthVersion, << "cAN8XSFAtQ98DdNbAKta8iJSRpw=" >>}]}]}]} [json_rpc:debug, 2018 - 12 -27T01: 00 : 34.996 - 08 : 00 ,ns_1 @172 .23. 121.17 :json_rpc_connection-goxdcr-cbauth< 0.23590 . 4 >:json_rpc_connection:handle_info: 94 ]got response: [{<< "id" >>, 0 }, {<< "result" >>, null }, {<< "error" >>, << "json: cannot unmarshal string into Go struct field Node.Ports of type int" >>}] Will put out a patch soon.

          Build couchbase-server-6.0.1-2014 contains ns_server commit abd76af with commit message:
          MB-32447: Add eventing debug port to node info...

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.1-2014 contains ns_server commit abd76af with commit message: MB-32447 : Add eventing debug port to node info...

          Build couchbase-server-6.5.0-1942 contains ns_server commit 3bcdd61 with commit message:
          Merge remote-tracking branch 'couchbase/alice'.

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1942 contains ns_server commit 3bcdd61 with commit message: Merge remote-tracking branch 'couchbase/alice'.

          Build couchbase-server-6.5.0-1942 contains ns_server commit abd76af with commit message:
          MB-32447: Add eventing debug port to node info...

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1942 contains ns_server commit abd76af with commit message: MB-32447 : Add eventing debug port to node info...

          verified for 5.5.0 , 5.5.1 ,5.1.0, 4.6.4 and 4.5.1 

          vikas.chaudhary Vikas Chaudhary added a comment - verified for 5.5.0 , 5.5.1 ,5.1.0, 4.6.4 and 4.5.1 

          People

            ajit.yagaty Ajit Yagaty [X] (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty