Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49991

[TLS] Rebalance exited with reason {service_rebalance_failed,fts, {agent_died, ...

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.6.5
    • 6.6.5
    • fts
    • 6.6.4-9946
    • Untriaged
    • 1
    • Yes

    Description

      To repro:

      • Setup single node with kv,index,n1ql,fts
      • enable TLS strict mode
      • load travel-sample
      • add new node (kv,index,n1ql,fts)
      • rebalance

      Got following error from UI:

      Rebalance exited with reason {service_rebalance_failed,fts, {agent_died,<27929.21642.0>, {noproc, {gen_server,call, [<27929.6905.1>, {call,\"ServiceAPI.GetTaskList\", #Fun<json_rpc_connection.0.23897766>}, 60000]}}}} 

      From log also see:

      Service 'fts' exited with status 1. Restarting. Messages:
      2021-12-08T10:15:16.762-08:00 [WARN] janitor: JanitorOnce, err: janitor: skipped on nil planPIndexes -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:98 

      Attachments

        1. image-2021-12-12-10-49-16-141.png
          101 kB
          Pierre Regazzoni
        2. image-2021-12-12-12-16-29-655.png
          153 kB
          Pierre Regazzoni
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          This is the reason why the rebalance failed ..

          2021-12-08T10:18:12.013-08:00 [FATA] init_http: listen, err: listen tcp [::1]:8094: bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go:252 

          abhinav Abhinav Dangeti added a comment - This is the reason why the rebalance failed .. 2021 - 12 -08T10: 18 : 12.013 - 08 : 00 [FATA] init_http: listen, err: listen tcp [:: 1 ]: 8094 : bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go: 252

          While on node 172.23.104.90, the listener was set up successfully at the IPv6 loopback ..

          2021-12-08T10:00:53.609-08:00 [INFO] init_http: web UI / REST API is available: http://[::1]:8094
          2021-12-08T10:00:53.609-08:00 [INFO] main: ------------------------------------------------------
          2021-12-08T10:00:53.609-08:00 [INFO] init_http: Setting up a http limit listener at "[::1]:8094" 

          On node 172.23.104.91, the listener fails ..

          2021-12-08T10:02:46.588-08:00 [INFO] init_http: web UI / REST API is available: http://[::1]:8094
          2021-12-08T10:02:46.588-08:00 [INFO] main: ------------------------------------------------------
          2021-12-08T10:02:46.588-08:00 [FATA] init_http: listen, err: listen tcp [::1]:8094: bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go:252

          Either why, we should be setting up ipv6 listeners only when enabled by user ..

          https://review.couchbase.org/c/cbft/+/167353

          abhinav Abhinav Dangeti added a comment - While on node 172.23.104.90, the listener was set up successfully at the IPv6 loopback .. 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] init_http: web UI / REST API is available: http: //[::1]:8094 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] main: ------------------------------------------------------ 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] init_http: Setting up a http limit listener at "[::1]:8094" On node 172.23.104.91, the listener fails .. 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [INFO] init_http: web UI / REST API is available: http: //[::1]:8094 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [INFO] main: ------------------------------------------------------ 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [FATA] init_http: listen, err: listen tcp [:: 1 ]: 8094 : bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go: 252 Either why, we should be setting up ipv6 listeners only when enabled by user .. https://review.couchbase.org/c/cbft/+/167353

          Wayne Siu would you approve this ticket?

          This was a regression introduced by changes made for MB-48044 (6.6.4).

          abhinav Abhinav Dangeti added a comment - Wayne Siu would you approve this ticket? This was a regression introduced by changes made for MB-48044 (6.6.4).
          wayne Wayne Siu added a comment -

          Re-opening it for 6.6.5.

          wayne Wayne Siu added a comment - Re-opening it for 6.6.5.

          Build 6.6.5-10060 contained cbft commit: 3790204597e681e87f87dfb99d074ab5aca2a912{}
          MB-49991: Behavior when non-SSL ports are disabled

          ceej Chris Hillery added a comment - Build 6.6.5-10060 contained cbft commit: 3790204597e681e87f87dfb99d074ab5aca2a912 { } MB-49991 : Behavior when non-SSL ports are disabled

          Verified on 6.6.5-10060

          pierre.regazzoni Pierre Regazzoni added a comment - Verified on 6.6.5-10060

          People

            abhinav Abhinav Dangeti
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty