ThreadSanitizer: dlopen() + RPATH incompatibility

Background

Couchbase server (e.g. engine_testapp) makes use of dlopen() to load the engine and the testsuite. The runtime linker determines the search path to use by looking at the values of RPATH and RUNPATH in the executable (e.g. engine_testapp)

• RUNPATH is the older property, it is used by the executable and any other libraries the executable loads to locate dlopen()ed files.

• RPATH is the newer (and more secure) property - is is only used when the executable itself loads a library - i.e. it isn't inherited by opened libraries like RUNPATH.

(Summary, see `man dlopen` for full details of search order).

CMake will set RPATH / RUNPATH (via linker arg -Wl) to the set of directories where all dependancies reside - and this is necessary for engine_testapp to load the engine and testsuite.

Problem

When running under ThreadSanitizer, TSan intercepts dlopen() and related functions, which results in the dlopen() appearing to come from libtsan.so. Given the above, this means that if RPATH is used, then the dlopen() for engine and testsuite fails, as libtsan doesn't have the path to ep.so for example embedded in it, and with RPATH the
paths arn't inherited from the main executable.

Newer versions of ld (at least Ubuntu 17.10) now use RPATH by default (as it is more secure), which means that we hit the above problem.