Delete UDS Sockets once they become stale

Problem Statement

When the js-evaluator or the eventing-consumer C++ process is killed, the UDS files that were created for communicating with it are not cleared. This behaviour leads to multiple .sock files accumulating in the /tmp/ folder.

This can become a security issue, in the following scenario:

• Suppose we have malicious client code that continuously causes the C++ process (i.e., V8) to crash, then every time the C++ process respawns it will use a new pair of UDS files leaving the old ones in the /tmp/ folder.
• This allows a malicious user to control the number of files created and residing on the filesystem, which is undesirable.

To ensure we maintain high levels of security we must prevent the accumulation of stale UDS files in the filesystem, i.e., in the /tmp/ folder in our case.

Note:

1. This problem is specific to Linux systems where we use UDS files.
2. This problem does not affect MacOS and Windows systems because they use TCP sockets.

Steps to Reproduce the Problem

1. Set up a Couchbase Server Cluster with at least one instance of the Eventing/Query service.
2. Observe the number of .sock files in the folder /tmp/
3. Send a kill signal to the eventing-consumer process in case of Eventing and the js-evaluator process in case of Query.
4. Observe the number of .sock files in the folder /tmp/
   1. You will notice that the number of .sock files has increased.
5. You can repeat the process from step 3 onwards to observe a further increase in the number of .sock files present in the /tmp/ folder.

Solution

• We must ensure we clear the old pair of UDS socket files before creating the new pair of socket files.
• To keep track of the old UDS socket files to delete, we can either:
  • Introduce bookkeeping in the Go layer (e.g., eventing-producer) to keep track of the paths to the UDS files, and delete them when a new pair is created.
    OR
  • Name the files according to a timestamp (e.g., seconds since the epoch) and then periodically clear files which are older than the latest two UDS files.
    • The idea is that the latest two UDS files would be part of the latest UDS file pair, currently in use for communication between the Go and C++ layers.
      OR
  • [Selected Solution] Introduce the code to clean up the stale UDS file(s) in code paths executed during the startup, restart and shutdown of the process so that whenever the process starts, restarts or shuts down, the stale UDS files will be immediately cleared.
    • When a healthy process is alive and executing requests/commands, no stale UDS files are created and hence no cleanup is required.