Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48211

[BP 7.0.2] XDCR - File descriptor leak in XDCR

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.6.0, 6.6.1, 6.6.2, 7.0.0, 6.6.3, 7.0.1
    • 7.0.2
    • XDCR
    • Untriaged
    • 1
    • No

    Description

      In a recent case from the field we've seen XDCR holding 70,000 sockets that do not have a process on the other side of the connection. When lsof is run, these sockets show up as follows:

       
      COMMAND     PID    USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
      ...
      goxdcr.bi 24440 cbadmin    5u     sock       0,7      0t0 303953540 protocol: TCP
      goxdcr.bi 24440 cbadmin    6u     sock       0,7      0t0 211821560 protocol: TCP
      goxdcr.bi 24440 cbadmin   10u     sock       0,7      0t0 216966092 protocol: TCP
      ...
      

      In this case the user had set the file descriptor limit to 70k and so at this point XDCR is unable to create new connections. This issue was previously tracked in MB-44182 and believed to be fixed, but it seems that the core issue hasn't been completely fixed.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-48211
          # Subject Branch Project Status CR V

          Activity

            Consistent number of file descriptors seen in 7.0.2-6683 longevity test. Verified using:
            while (( 1 == 1 ));do lsof -p `ps -eaf | grep goxdcr | grep 8091 | awk '

            {print $2}

            '` | awk '

            {print $1}

            ' | grep goxdcr | sort | uniq -c | sort -rn;done

            pavithra.mahamani Pavithra Mahamani added a comment - Consistent number of file descriptors seen in 7.0.2-6683 longevity test. Verified using: while (( 1 == 1 ));do lsof -p `ps -eaf | grep goxdcr | grep 8091 | awk ' {print $2} '` | awk ' {print $1} ' | grep goxdcr | sort | uniq -c | sort -rn;done
            wayne Wayne Siu added a comment -

            Neil Huang
            Is this ticket done or more changes are expected? Thanks.

            wayne Wayne Siu added a comment - Neil Huang Is this ticket done or more changes are expected? Thanks.

            Build couchbase-server-7.0.2-6613 contains goxdcr commit 4d2e022 with commit message:
            MB-48211 - XDCR file descriptor leak when system is busy

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6613 contains goxdcr commit 4d2e022 with commit message: MB-48211 - XDCR file descriptor leak when system is busy

            People

              pavithra.mahamani Pavithra Mahamani
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty