Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49991

[TLS] Rebalance exited with reason {service_rebalance_failed,fts, {agent_died, ...

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.6.5
    • 6.6.5
    • fts
    • 6.6.4-9946
    • Untriaged
    • 1
    • Yes

    Description

      To repro:

      • Setup single node with kv,index,n1ql,fts
      • enable TLS strict mode
      • load travel-sample
      • add new node (kv,index,n1ql,fts)
      • rebalance

      Got following error from UI:

      Rebalance exited with reason {service_rebalance_failed,fts, {agent_died,<27929.21642.0>, {noproc, {gen_server,call, [<27929.6905.1>, {call,\"ServiceAPI.GetTaskList\", #Fun<json_rpc_connection.0.23897766>}, 60000]}}}} 

      From log also see:

      Service 'fts' exited with status 1. Restarting. Messages:
      2021-12-08T10:15:16.762-08:00 [WARN] janitor: JanitorOnce, err: janitor: skipped on nil planPIndexes -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:98 

      Attachments

        For Gerrit Dashboard: MB-49991
        # Subject Branch Project Status CR V

        Activity

          pierre.regazzoni Pierre Regazzoni created issue -
          mihir.kamdar Mihir Kamdar (Inactive) made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Critical [ 2 ]
          abhinav Abhinav Dangeti made changes -
          Assignee Keshav Murthy [ keshav ] Abhinav Dangeti [ abhinav ]

          This is the reason why the rebalance failed ..

          2021-12-08T10:18:12.013-08:00 [FATA] init_http: listen, err: listen tcp [::1]:8094: bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go:252 

          abhinav Abhinav Dangeti added a comment - This is the reason why the rebalance failed .. 2021 - 12 -08T10: 18 : 12.013 - 08 : 00 [FATA] init_http: listen, err: listen tcp [:: 1 ]: 8094 : bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go: 252

          While on node 172.23.104.90, the listener was set up successfully at the IPv6 loopback ..

          2021-12-08T10:00:53.609-08:00 [INFO] init_http: web UI / REST API is available: http://[::1]:8094
          2021-12-08T10:00:53.609-08:00 [INFO] main: ------------------------------------------------------
          2021-12-08T10:00:53.609-08:00 [INFO] init_http: Setting up a http limit listener at "[::1]:8094" 

          On node 172.23.104.91, the listener fails ..

          2021-12-08T10:02:46.588-08:00 [INFO] init_http: web UI / REST API is available: http://[::1]:8094
          2021-12-08T10:02:46.588-08:00 [INFO] main: ------------------------------------------------------
          2021-12-08T10:02:46.588-08:00 [FATA] init_http: listen, err: listen tcp [::1]:8094: bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go:252

          Either why, we should be setting up ipv6 listeners only when enabled by user ..

          https://review.couchbase.org/c/cbft/+/167353

          abhinav Abhinav Dangeti added a comment - While on node 172.23.104.90, the listener was set up successfully at the IPv6 loopback .. 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] init_http: web UI / REST API is available: http: //[::1]:8094 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] main: ------------------------------------------------------ 2021 - 12 -08T10: 00 : 53.609 - 08 : 00 [INFO] init_http: Setting up a http limit listener at "[::1]:8094" On node 172.23.104.91, the listener fails .. 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [INFO] init_http: web UI / REST API is available: http: //[::1]:8094 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [INFO] main: ------------------------------------------------------ 2021 - 12 -08T10: 02 : 46.588 - 08 : 00 [FATA] init_http: listen, err: listen tcp [:: 1 ]: 8094 : bind: cannot assign requested address -- main.mainServeHTTP() at init_http.go: 252 Either why, we should be setting up ipv6 listeners only when enabled by user .. https://review.couchbase.org/c/cbft/+/167353

          Wayne Siu would you approve this ticket?

          This was a regression introduced by changes made for MB-48044 (6.6.4).

          abhinav Abhinav Dangeti added a comment - Wayne Siu would you approve this ticket? This was a regression introduced by changes made for MB-48044 (6.6.4).
          abhinav Abhinav Dangeti made changes -
          Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
          wayne Wayne Siu made changes -
          Labels TLS 6.6.4-candidate TLS
          abhinav Abhinav Dangeti made changes -
          Link This issue blocks MB-47673 [ MB-47673 ]
          abhinav Abhinav Dangeti made changes -
          Assignee Abhinav Dangeti [ abhinav ] Pierre Regazzoni [ JIRAUSER25157 ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          wayne Wayne Siu made changes -
          Link This issue blocks MB-50048 [ MB-50048 ]
          wayne Wayne Siu made changes -
          Labels 6.6.4-candidate TLS 6.6.4-candidate TLS approved-for-6.6.5
          wayne Wayne Siu made changes -
          Labels 6.6.4-candidate TLS approved-for-6.6.5 TLS approved-for-6.6.5
          wayne Wayne Siu made changes -
          Link This issue is caused by MB-48044 [ MB-48044 ]
          wayne Wayne Siu made changes -
          Link This issue relates to MB-48997 [ MB-48997 ]
          wayne Wayne Siu added a comment -

          Re-opening it for 6.6.5.

          wayne Wayne Siu added a comment - Re-opening it for 6.6.5.
          wayne Wayne Siu made changes -
          Assignee Pierre Regazzoni [ JIRAUSER25157 ] Abhinav Dangeti [ abhinav ]
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]

          Build 6.6.5-10060 contained cbft commit: 3790204597e681e87f87dfb99d074ab5aca2a912{}
          MB-49991: Behavior when non-SSL ports are disabled

          ceej Chris Hillery added a comment - Build 6.6.5-10060 contained cbft commit: 3790204597e681e87f87dfb99d074ab5aca2a912 { } MB-49991 : Behavior when non-SSL ports are disabled
          pierre.regazzoni Pierre Regazzoni made changes -
          Attachment image-2021-12-12-10-49-16-141.png [ 172972 ]
          pierre.regazzoni Pierre Regazzoni made changes -
          Attachment image-2021-12-12-12-16-29-655.png [ 172973 ]

          Verified on 6.6.5-10060

          pierre.regazzoni Pierre Regazzoni added a comment - Verified on 6.6.5-10060
          abhinav Abhinav Dangeti made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          girish.benakappa Girish Benakappa made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          wayne Wayne Siu made changes -
          Link This issue is cloned by MB-51388 [ MB-51388 ]
          wayne Wayne Siu made changes -
          Link This issue relates to MB-48997 [ MB-48997 ]
          abhinav Abhinav Dangeti made changes -
          Link This issue is duplicated by MB-51388 [ MB-51388 ]
          abhinav Abhinav Dangeti made changes -
          Link This issue is cloned by MB-51388 [ MB-51388 ]

          People

            abhinav Abhinav Dangeti
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty