Details
-
Task
-
Resolution: Fixed
-
Major
-
6.0.5, 6.6.0
-
1
-
KV-Engine Sprint 2020-Dec
Description
After CV workers were upgraded to ubuntu18.04, memcached_memory_tracking_test fails.
[ RUN ] MemoryTrackerTest.Accounting
|
terminate called after throwing an instance of 'std::runtime_error'
|
what(): memcached_memory_tracking_plugin.so: cannot open shared object file: No such file or directory
|
The so is present and correct, in build/kv_engine.
memcached_memory_tracking_test has the correct directory added to RUNPATH
couchbase@df22eb15ad83:~/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/kv_engine$ readelf -d ./memcached_memory_tracking_test
|
|
|
Dynamic section at offset 0x5bd48 contains 37 entries:
|
Tag Type Name/Value
|
0x0000000000000001 (NEEDED) Shared library: [libmemcached_logger.so.1.0.0]
|
0x0000000000000001 (NEEDED) Shared library: [libjemalloc.so.2]
|
0x0000000000000001 (NEEDED) Shared library: [libdirutils.so.0.1.0]
|
0x0000000000000001 (NEEDED) Shared library: [libplatform_so.so.0.1.0]
|
0x0000000000000001 (NEEDED) Shared library: [libevent_core.so.2.1.8]
|
0x0000000000000001 (NEEDED) Shared library: [libevent_extra.so.2.1.8]
|
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
|
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
|
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
|
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
|
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
|
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
|
0x000000000000001d (RUNPATH) Library runpath: [/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/kv_engine:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/phosphor]
|
0x000000000000000c (INIT) 0x410d98
|
0x000000000000000d (FINI) 0x447144
|
0x0000000000000019 (INIT_ARRAY) 0x65b8e0
|
0x000000000000001b (INIT_ARRAYSZ) 24 (bytes)
|
0x000000000000001a (FINI_ARRAY) 0x65b8f8
|
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
|
0x0000000000000004 (HASH) 0x4002b0
|
0x000000006ffffef5 (GNU_HASH) 0x401620
|
0x0000000000000005 (STRTAB) 0x406c00
|
0x0000000000000006 (SYMTAB) 0x402880
|
0x000000000000000a (STRSZ) 34359 (bytes)
|
0x000000000000000b (SYMENT) 24 (bytes)
|
0x0000000000000015 (DEBUG) 0x0
|
0x0000000000000003 (PLTGOT) 0x65c000
|
0x0000000000000002 (PLTRELSZ) 4800 (bytes)
|
0x0000000000000014 (PLTREL) RELA
|
0x0000000000000017 (JMPREL) 0x40fad8
|
0x0000000000000007 (RELA) 0x40f8f8
|
0x0000000000000008 (RELASZ) 480 (bytes)
|
0x0000000000000009 (RELAENT) 24 (bytes)
|
0x000000006ffffffe (VERNEED) 0x40f7d8
|
0x000000006fffffff (VERNEEDNUM) 4
|
0x000000006ffffff0 (VERSYM) 0x40f238
|
0x0000000000000000 (NULL) 0x0
|
But the actual dlopen call is made from within libplatform_so.so, by cb::io::loadLibrary. With LD_DEBUG=libs, it is apparent the RUNPATH from platform is searched.
78089: find library=memcached_memory_tracking_plugin [0]; searching
|
78089: search path=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib (RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/libplatform_so.so.0.1.0)
|
78089: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib/memcached_memory_tracking_plugin
|
78089: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/memcached_memory_tracking_plugin
|
78089: search cache=/etc/ld.so.cache
|
78089: search path=/lib/x86_64-linux-gnu/tls/x86_64/x86_64:/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls:/lib/x86_64-linux-gnu/x86_64/x86_64:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/tls/x86_64/x86_64:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls:/usr/lib/x86_64-linux-gnu/x86_64/x86_64:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu:/lib/tls/x86_64/x86_64:/lib/tls/x86_64:/lib/tls/x86_64:/lib/tls:/lib/x86_64/x86_64:/lib/x86_64:/lib/x86_64:/lib:/usr/lib/tls/x86_64/x86_64:/usr/lib/tls/x86_64:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64/x86_64:/usr/lib/x86_64:/usr/lib/x86_64:/usr/lib (system search path)
|
78089: trying file=/lib/x86_64-linux-gnu/tls/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/tls/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/tls/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin
|
78089: trying file=/lib/tls/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/tls/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/lib/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/tls/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/tls/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/tls/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/x86_64/memcached_memory_tracking_plugin
|
78089: trying file=/usr/lib/memcached_memory_tracking_plugin
|
78089:
|
78089: find library=memcached_memory_tracking_plugin.so [0]; searching
|
78089: search path=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib (RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/libplatform_so.so.0.1.0)
|
78089: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib/memcached_memory_tracking_plugin.so
|
78089: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/memcached_memory_tracking_plugin.so
|
78089: search cache=/etc/ld.so.cache
|
78089: search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib:/usr/lib (system search path)
|
78089: trying file=/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin.so
|
78089: trying file=/usr/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin.so
|
78089: trying file=/lib/memcached_memory_tracking_plugin.so
|
78089: trying file=/usr/lib/memcached_memory_tracking_plugin.so
|
Prior to the upgrade, the linker did not by default set RUNPATH (but could be caused to do so with --enable-new-dtags), but instead set RPATH. Now, RUNPATH is used by default.
So,
link_directories(${Memcached_BINARY_DIR})
|
would previously would add build/kv_engine to RPATH, but now adds it to RUNPATH.
With
target_link_options(memcached_memory_tracking_test PRIVATE LINKER:--disable-new-dtags)
|
the "old" behaviour can be applied just for that executable. build/kv_engine is set in RPATH, and no RUNPATH is present
0x000000000000000f (RPATH) Library rpath: [/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/kv_engine:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib]
|
And yet with LD_DEBUG=libs, while memcached_memory_tracking_test RPATH is noted, build/kv_engine is still not searched during dlopen.
[ RUN ] MemoryTrackerTest.Accounting
|
117151: find library=memcached_memory_tracking_plugin [0]; searching
|
117151: search path=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib (RPATH from file ./build/kv_engine/memcached_memory_tracking_test)
|
117151: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform/memcached_memory_tracking_plugin
|
117151: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor/memcached_memory_tracking_plugin
|
117151: trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib/memcached_memory_tracking_plugin
|
117151: search path=tls/x86_64/x86_64:tls/x86_64:tls/x86_64:tls:x86_64/x86_64:x86_64:x86_64: (RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform/libplatform_so.so.0.1.0)
|
Applying --disable-new-dtags globally would likely resolve this, but would be rather heavy handed.
This does not cause issues on master, as cb::io::loadLibrary has been removed, as has the failing part of the test.