Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42358

memcached_memory_tracking_test fails after CV upgrade to ubuntu18.04

    XMLWordPrintable

Details

    • 1
    • KV-Engine Sprint 2020-Dec

    Description

      After CV workers were upgraded to ubuntu18.04, memcached_memory_tracking_test fails.

       [ RUN      ] MemoryTrackerTest.Accounting
       terminate called after throwing an instance of 'std::runtime_error'
         what():  memcached_memory_tracking_plugin.so: cannot open shared object file: No such file or directory
      

      The so is present and correct, in build/kv_engine.

      memcached_memory_tracking_test has the correct directory added to RUNPATH

      couchbase@df22eb15ad83:~/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/kv_engine$ readelf -d ./memcached_memory_tracking_test
       
      Dynamic section at offset 0x5bd48 contains 37 entries:
        Tag        Type                         Name/Value
       0x0000000000000001 (NEEDED)             Shared library: [libmemcached_logger.so.1.0.0]
       0x0000000000000001 (NEEDED)             Shared library: [libjemalloc.so.2]
       0x0000000000000001 (NEEDED)             Shared library: [libdirutils.so.0.1.0]
       0x0000000000000001 (NEEDED)             Shared library: [libplatform_so.so.0.1.0]
       0x0000000000000001 (NEEDED)             Shared library: [libevent_core.so.2.1.8]
       0x0000000000000001 (NEEDED)             Shared library: [libevent_extra.so.2.1.8]
       0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
       0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
       0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
       0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
       0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
       0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
       0x000000000000001d (RUNPATH)            Library runpath: [/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/kv_engine:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/build/phosphor]
       0x000000000000000c (INIT)               0x410d98
       0x000000000000000d (FINI)               0x447144
       0x0000000000000019 (INIT_ARRAY)         0x65b8e0
       0x000000000000001b (INIT_ARRAYSZ)       24 (bytes)
       0x000000000000001a (FINI_ARRAY)         0x65b8f8
       0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
       0x0000000000000004 (HASH)               0x4002b0
       0x000000006ffffef5 (GNU_HASH)           0x401620
       0x0000000000000005 (STRTAB)             0x406c00
       0x0000000000000006 (SYMTAB)             0x402880
       0x000000000000000a (STRSZ)              34359 (bytes)
       0x000000000000000b (SYMENT)             24 (bytes)
       0x0000000000000015 (DEBUG)              0x0
       0x0000000000000003 (PLTGOT)             0x65c000
       0x0000000000000002 (PLTRELSZ)           4800 (bytes)
       0x0000000000000014 (PLTREL)             RELA
       0x0000000000000017 (JMPREL)             0x40fad8
       0x0000000000000007 (RELA)               0x40f8f8
       0x0000000000000008 (RELASZ)             480 (bytes)
       0x0000000000000009 (RELAENT)            24 (bytes)
       0x000000006ffffffe (VERNEED)            0x40f7d8
       0x000000006fffffff (VERNEEDNUM)         4
       0x000000006ffffff0 (VERSYM)             0x40f238
       0x0000000000000000 (NULL)               0x0
      

      But the actual dlopen call is made from within libplatform_so.so, by cb::io::loadLibrary. With LD_DEBUG=libs, it is apparent the RUNPATH from platform is searched.

      78089:     find library=memcached_memory_tracking_plugin [0]; searching
      78089:      search path=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib             (RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/libplatform_so.so.0.1.0)
      78089:       trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib/memcached_memory_tracking_plugin
      78089:       trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/memcached_memory_tracking_plugin
      78089:      search cache=/etc/ld.so.cache
      78089:      search path=/lib/x86_64-linux-gnu/tls/x86_64/x86_64:/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls:/lib/x86_64-linux-gnu/x86_64/x86_64:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/tls/x86_64/x86_64:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls:/usr/lib/x86_64-linux-gnu/x86_64/x86_64:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu:/lib/tls/x86_64/x86_64:/lib/tls/x86_64:/lib/tls/x86_64:/lib/tls:/lib/x86_64/x86_64:/lib/x86_64:/lib/x86_64:/lib:/usr/lib/tls/x86_64/x86_64:/usr/lib/tls/x86_64:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64/x86_64:/usr/lib/x86_64:/usr/lib/x86_64:/usr/lib            (system search path)
      78089:       trying file=/lib/x86_64-linux-gnu/tls/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/tls/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/tls/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin
      78089:       trying file=/lib/tls/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/tls/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/lib/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/tls/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/tls/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/tls/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/x86_64/memcached_memory_tracking_plugin
      78089:       trying file=/usr/lib/memcached_memory_tracking_plugin
      78089:
      78089:     find library=memcached_memory_tracking_plugin.so [0]; searching
      78089:      search path=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib:/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib             (RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/libplatform_so.so.0.1.0)
      78089:       trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/../lib/memcached_memory_tracking_plugin.so
      78089:       trying file=/home/couchbase/jenkins/workspace/kv_engine.linux-CE_mad-hatter/install/lib/memcached_memory_tracking_plugin.so
      78089:      search cache=/etc/ld.so.cache
      78089:      search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib:/usr/lib              (system search path)
      78089:       trying file=/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin.so
      78089:       trying file=/usr/lib/x86_64-linux-gnu/memcached_memory_tracking_plugin.so
      78089:       trying file=/lib/memcached_memory_tracking_plugin.so
      78089:       trying file=/usr/lib/memcached_memory_tracking_plugin.so
      

      Prior to the upgrade, the linker did not by default set RUNPATH (but could be caused to do so with --enable-new-dtags), but instead set RPATH. Now, RUNPATH is used by default.

      So,

      link_directories(${Memcached_BINARY_DIR})
      

      would previously would add build/kv_engine to RPATH, but now adds it to RUNPATH.

      With

      target_link_options(memcached_memory_tracking_test PRIVATE LINKER:--disable-new-dtags)
      

      the "old" behaviour can be applied just for that executable. build/kv_engine is set in RPATH, and no RUNPATH is present

      0x000000000000000f (RPATH)              Library rpath: [/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/kv_engine:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib]
      

      And yet with LD_DEBUG=libs, while memcached_memory_tracking_test RPATH is noted, build/kv_engine is still not searched during dlopen.

      [ RUN      ] MemoryTrackerTest.Accounting
          117151:	find library=memcached_memory_tracking_plugin [0]; searching
          117151:	 search path=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor:/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib		(RPATH from file ./build/kv_engine/memcached_memory_tracking_test)
          117151:	  trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform/memcached_memory_tracking_plugin
          117151:	  trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/phosphor/memcached_memory_tracking_plugin
          117151:	  trying file=/home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/install/lib/memcached_memory_tracking_plugin
          117151:	 search path=tls/x86_64/x86_64:tls/x86_64:tls/x86_64:tls:x86_64/x86_64:x86_64:x86_64:		(RUNPATH from file /home/couchbase/jenkins/workspace/kv_engine.linux_mad-hatter/build/platform/libplatform_so.so.0.1.0)
      

      Applying --disable-new-dtags globally would likely resolve this, but would be rather heavy handed.

      This does not cause issues on master, as cb::io::loadLibrary has been removed, as has the failing part of the test.

      Attachments

        For Gerrit Dashboard: MB-42358
        # Subject Branch Project Status CR V

        Activity

          Build couchbase-server-7.0.0-3841 contains kv_engine commit c45e701 with commit message:
          MB-42358: Pass path to memory-tracking test plugin on command-line

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3841 contains kv_engine commit c45e701 with commit message: MB-42358 : Pass path to memory-tracking test plugin on command-line
          paolo.cocchi Paolo Cocchi added a comment -

          Re-opening for backport to 6.5.2.

          paolo.cocchi Paolo Cocchi added a comment - Re-opening for backport to 6.5.2.

          Build couchbase-server-6.5.2-6603 contains kv_engine commit 880737e with commit message:
          MB-42358 [BP]: Pass path to memory-tracking test plugin on command-line

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.2-6603 contains kv_engine commit 880737e with commit message: MB-42358 [BP] : Pass path to memory-tracking test plugin on command-line

          Build couchbase-server-6.6.2-9417 contains kv_engine commit 880737e with commit message:
          MB-42358 [BP]: Pass path to memory-tracking test plugin on command-line

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9417 contains kv_engine commit 880737e with commit message: MB-42358 [BP] : Pass path to memory-tracking test plugin on command-line

          Build couchbase-server-7.0.0-4255 contains kv_engine commit 880737e with commit message:
          MB-42358 [BP]: Pass path to memory-tracking test plugin on command-line

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4255 contains kv_engine commit 880737e with commit message: MB-42358 [BP] : Pass path to memory-tracking test plugin on command-line

          People

            paolo.cocchi Paolo Cocchi
            james.harrison James Harrison
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                PagerDuty