Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21474

[FTS] queries fail under low-amount of concurrent load on multi-node cluster due to too many connections

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      cbft opens tcp connections between nodes but doesn't reuse them.
      Amount of connections quickly reaches the limit and FTS queries start returning connection errors:

      "status": {
      "errors": {
      "IndexClient - http://172.23.99.39:8094/api/pindex/perf_fts_index_420d95bfb1812a98_321cbb28": "Post http://%40fts-cbauth:3e9bc4ea6b573dd5fefe64150f9bc385@172.23.99.39:8094/api/pindex/perf_fts_index_420d95bfb1812a98_321cbb28/query: dial tcp 172.23.99.39:8094: connect: cannot assign requested address",
      .

      netstat output:

      [root@fts01-s424 ~]# netstat -lpnta | grep "TIME_WAIT - " | wc -l
      55774

      [root@fts01-s424 ~]# netstat -lpnta
      tcp 0 0 172.23.99.211:51190 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:34465 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:57708 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:47835 172.23.99.40:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:33509 172.23.99.40:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:54078 172.23.99.40:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:47845 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:57334 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:51675 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:39886 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:60856 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:50033 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:49765 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:40220 172.23.99.39:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:49538 172.23.99.40:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:46047 172.23.99.40:8094 TIME_WAIT -
      tcp 0 0 172.23.99.211:34501 172.23.99.39:8094 TIME_WAIT -
      ...

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-21474
          # Subject Branch Project Status CR V

          Activity

            People

              steve Steve Yen
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty