[MB-49881] libstdc++ debug file is missing from RPM - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 7.0.0, 7.1.0
Fix Version/s: 7.1.0, 7.0.4
Component/s: build
Labels:
Environment:
7.1.0-1817.x86_64
CentOS Linux release 7.8.2003 (Core)

Triage:
Untriaged
Story Points:
1
Is this a Regression?:
Unknown
Sprint:
Build Team 2021 Sprint 24, Build Team 2021 Sprint 25, Build Team 2022 Sprint 1, Build Team 2022 Sprint 2

Description

When trying to debug a memcached issue on CB Server 7.1.0-1817 on CentOS 7, GDB is reporting that debugging information is missing even when the -debuginfo RPM is installed:

rpm -qa|grep couchbase

couchbase-server-debuginfo-7.1.0-1817.x86_64

couchbase-server-7.1.0-1817.x86_64

Observed Behaviour

GDB cannot find debugging information for our version of the C++ standard library /opt/couchbase/bin/../lib/libstdc++.so.6:

gdb -p $(pgrep memcached)

...

(gdb) info sharedlibrary

From                To                  Syms Read   Shared Object Library

0x00007f0d19a102c0  0x00007f0d19a1493e  Yes         /opt/couchbase/bin/../lib/libnuma.so.1

0x00007f0d19809fe0  0x00007f0d1980abfa  Yes         /opt/couchbase/bin/../lib/liburing.so.1

0x00007f0d195ee290  0x00007f0d19601670  Yes (*)     /opt/couchbase/bin/../lib/libz.so.1

0x00007f0d193d0030  0x00007f0d193e2455  Yes         /opt/couchbase/bin/../lib/libphosphor.so

0x00007f0d191c3e50  0x00007f0d191c494e  Yes         /lib64/libdl.so.2

0x00007f0d18fac8f0  0x00007f0d18fb7db1  Yes         /lib64/libpthread.so.0

0x00007f0d18da1230  0x00007f0d18da41bc  Yes         /lib64/librt.so.1

0x00007f0d18b71b90  0x00007f0d18b9b3a7  Yes (*)     /opt/couchbase/bin/../lib/liblz4.so.1

0x00007f0d189675c0  0x00007f0d1896c97f  Yes         /opt/couchbase/bin/../lib/libsnappy.so.1

0x00007f0d18495750  0x00007f0d1852abfa  Yes         /opt/couchbase/bin/../lib/libjemalloc.so.2

0x00007f0d1826a2f0  0x00007f0d182836c1  Yes         /opt/couchbase/bin/../lib/libevent_core-2.1.so.7

0x00007f0d18048e90  0x00007f0d1805961f  Yes         /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7

0x00007f0d17e3fd50  0x00007f0d17e400f7  Yes         /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7

0x00007f0d17c3a6f0  0x00007f0d17c3caa4  Yes         /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7

0x00007f0d179c20d0  0x00007f0d17a0fb16  Yes (*)     /opt/couchbase/bin/../lib/libssl.so.1.1

0x00007f0d1751d000  0x00007f0d176dbf4a  Yes (*)     /opt/couchbase/bin/../lib/libcrypto.so.1.1

0x00007f0d17171ef0  0x00007f0d172498ba  Yes (*)     /opt/couchbase/bin/../lib/libstdc++.so.6

0x00007f0d16ddc350  0x00007f0d16e47316  Yes         /lib64/libm.so.6

0x00007f0d16bc1af0  0x00007f0d16bd22dd  Yes (*)     /opt/couchbase/bin/../lib/libgcc_s.so.1

0x00007f0d168109f0  0x00007f0d1696093f  Yes         /lib64/libc.so.6

0x00007f0d19c18af0  0x00007f0d19c34050  Yes         /lib64/ld-linux-x86-64.so.2

(*): Shared library is missing debugging information.

Expected behaviour

Like our other libraries (e.g. /opt/couchbase/bin/../lib/libjemalloc.so.2) debugging information should be present.

Initial analysis

The expected .debug file is missing from the RPM - for example here are all "lib" prefixed debug files in the lib directory - note no libstdc++:

# rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep /usr/lib/debug/opt/couchbase/lib/lib

/usr/lib/debug/opt/couchbase/lib/libbreakpad_wrapper.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libchrome_zlib.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libcouchbase.so.8.0.5-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libcouchstore.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libcouchstore_views.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libcurl.so.4.7.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libevent-2.1.so.7.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libforestdb.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libicui18n.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libicuuc.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libjseval.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libpcre.so.1.2.12-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libpcrecpp.so.0.0.2-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libpcreposix.so.0.0.7-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libphosphor.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libsigar.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/liburing.so.1.0.6-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libuv.so.1.0.0-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libv8.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libv8_libbase.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libv8_libplatform.so-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11-7.1.0-1817.x86_64.debug

(Seen while investigating ~~MB-49877~~).

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Dave Rigby created issue - 03/Dec/21 1:51 AM

Dave Rigby made changes - 03/Dec/21 1:53 AM

Field	Original Value	New Value
Description	When trying to debug a memcached issue on CB Server 7.1.0-1817 on CentOS 7, GDB is reporting that debugging information even when the -debuginfo RPM is installed: {code} rpm -qa\|grep couchbase couchbase-server-debuginfo-7.1.0-1817.x86_64 couchbase-server-7.1.0-1817.x86_64 {code} h3. Observed Behaviour GDB cannot find debugging information for our version of the C++ standard library {{/opt/couchbase/bin/../lib/libstdc++.so.6}}: {code} gdb -p $(pgrep memcached) ... (gdb) info sharedlibrary From To Syms Read Shared Object Library 0x00007f0d19a102c0 0x00007f0d19a1493e Yes /opt/couchbase/bin/../lib/libnuma.so.1 0x00007f0d19809fe0 0x00007f0d1980abfa Yes /opt/couchbase/bin/../lib/liburing.so.1 0x00007f0d195ee290 0x00007f0d19601670 Yes () /opt/couchbase/bin/../lib/libz.so.1 0x00007f0d193d0030 0x00007f0d193e2455 Yes /opt/couchbase/bin/../lib/libphosphor.so 0x00007f0d191c3e50 0x00007f0d191c494e Yes /lib64/libdl.so.2 0x00007f0d18fac8f0 0x00007f0d18fb7db1 Yes /lib64/libpthread.so.0 0x00007f0d18da1230 0x00007f0d18da41bc Yes /lib64/librt.so.1 0x00007f0d18b71b90 0x00007f0d18b9b3a7 Yes () /opt/couchbase/bin/../lib/liblz4.so.1 0x00007f0d189675c0 0x00007f0d1896c97f Yes /opt/couchbase/bin/../lib/libsnappy.so.1 0x00007f0d18495750 0x00007f0d1852abfa Yes /opt/couchbase/bin/../lib/libjemalloc.so.2 0x00007f0d1826a2f0 0x00007f0d182836c1 Yes /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 0x00007f0d18048e90 0x00007f0d1805961f Yes /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 0x00007f0d17e3fd50 0x00007f0d17e400f7 Yes /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 0x00007f0d17c3a6f0 0x00007f0d17c3caa4 Yes /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 0x00007f0d179c20d0 0x00007f0d17a0fb16 Yes () /opt/couchbase/bin/../lib/libssl.so.1.1 0x00007f0d1751d000 0x00007f0d176dbf4a Yes () /opt/couchbase/bin/../lib/libcrypto.so.1.1 0x00007f0d17171ef0 0x00007f0d172498ba Yes () /opt/couchbase/bin/../lib/libstdc++.so.6 0x00007f0d16ddc350 0x00007f0d16e47316 Yes /lib64/libm.so.6 0x00007f0d16bc1af0 0x00007f0d16bd22dd Yes () /opt/couchbase/bin/../lib/libgcc_s.so.1 0x00007f0d168109f0 0x00007f0d1696093f Yes /lib64/libc.so.6 0x00007f0d19c18af0 0x00007f0d19c34050 Yes /lib64/ld-linux-x86-64.so.2 (*): Shared library is missing debugging information. {code} h3. Expected behaviour Like our other libraries (e.g. /opt/couchbase/bin/../lib/libjemalloc.so.2) debugging information should be present. h3. Initial analysis The expected .debug file is missing from the RPM - for example here are all "lib" prefixed debug files in the lib directory - note no libstdc++: {code} # rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64\|grep /usr/lib/debug/opt/couchbase/lib/lib /usr/lib/debug/opt/couchbase/lib/libbreakpad_wrapper.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libchrome_zlib.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchbase.so.8.0.5-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchstore.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchstore_views.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcurl.so.4.7.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libforestdb.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libicui18n.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libicuuc.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libjseval.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcre.so.1.2.12-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcrecpp.so.0.0.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcreposix.so.0.0.7-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libphosphor.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libsigar.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/liburing.so.1.0.6-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libuv.so.1.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8_libbase.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8_libplatform.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11-7.1.0-1817.x86_64.debug {code} (Seen while investigating ~~MB-49877~~).	When trying to debug a memcached issue on CB Server 7.1.0-1817 on CentOS 7, GDB is reporting that debugging information even when the -debuginfo RPM is installed: {code} rpm -qa\|grep couchbase couchbase-server-debuginfo-7.1.0-1817.x86_64 couchbase-server-7.1.0-1817.x86_64 {code} h3. Observed Behaviour GDB cannot find debugging information for our version of the C++ standard library {{/opt/couchbase/bin/../lib/libstdc++.so.6}}: {code:highlight=21} gdb -p $(pgrep memcached) ... (gdb) info sharedlibrary From To Syms Read Shared Object Library 0x00007f0d19a102c0 0x00007f0d19a1493e Yes /opt/couchbase/bin/../lib/libnuma.so.1 0x00007f0d19809fe0 0x00007f0d1980abfa Yes /opt/couchbase/bin/../lib/liburing.so.1 0x00007f0d195ee290 0x00007f0d19601670 Yes () /opt/couchbase/bin/../lib/libz.so.1 0x00007f0d193d0030 0x00007f0d193e2455 Yes /opt/couchbase/bin/../lib/libphosphor.so 0x00007f0d191c3e50 0x00007f0d191c494e Yes /lib64/libdl.so.2 0x00007f0d18fac8f0 0x00007f0d18fb7db1 Yes /lib64/libpthread.so.0 0x00007f0d18da1230 0x00007f0d18da41bc Yes /lib64/librt.so.1 0x00007f0d18b71b90 0x00007f0d18b9b3a7 Yes () /opt/couchbase/bin/../lib/liblz4.so.1 0x00007f0d189675c0 0x00007f0d1896c97f Yes /opt/couchbase/bin/../lib/libsnappy.so.1 0x00007f0d18495750 0x00007f0d1852abfa Yes /opt/couchbase/bin/../lib/libjemalloc.so.2 0x00007f0d1826a2f0 0x00007f0d182836c1 Yes /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 0x00007f0d18048e90 0x00007f0d1805961f Yes /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 0x00007f0d17e3fd50 0x00007f0d17e400f7 Yes /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 0x00007f0d17c3a6f0 0x00007f0d17c3caa4 Yes /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 0x00007f0d179c20d0 0x00007f0d17a0fb16 Yes () /opt/couchbase/bin/../lib/libssl.so.1.1 0x00007f0d1751d000 0x00007f0d176dbf4a Yes () /opt/couchbase/bin/../lib/libcrypto.so.1.1 0x00007f0d17171ef0 0x00007f0d172498ba Yes () /opt/couchbase/bin/../lib/libstdc++.so.6 0x00007f0d16ddc350 0x00007f0d16e47316 Yes /lib64/libm.so.6 0x00007f0d16bc1af0 0x00007f0d16bd22dd Yes () /opt/couchbase/bin/../lib/libgcc_s.so.1 0x00007f0d168109f0 0x00007f0d1696093f Yes /lib64/libc.so.6 0x00007f0d19c18af0 0x00007f0d19c34050 Yes /lib64/ld-linux-x86-64.so.2 (*): Shared library is missing debugging information. {code} h3. Expected behaviour Like our other libraries (e.g. /opt/couchbase/bin/../lib/libjemalloc.so.2) debugging information should be present. h3. Initial analysis The expected .debug file is missing from the RPM - for example here are all "lib" prefixed debug files in the lib directory - note no libstdc++: {code} # rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64\|grep /usr/lib/debug/opt/couchbase/lib/lib /usr/lib/debug/opt/couchbase/lib/libbreakpad_wrapper.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libchrome_zlib.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchbase.so.8.0.5-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchstore.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcouchstore_views.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libcurl.so.4.7.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libforestdb.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libicui18n.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libicuuc.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libjseval.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcre.so.1.2.12-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcrecpp.so.0.0.2-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libpcreposix.so.0.0.7-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libphosphor.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libsigar.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/liburing.so.1.0.6-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libuv.so.1.0.0-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8_libbase.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libv8_libplatform.so-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11-7.1.0-1817.x86_64.debug {code} (Seen while investigating ~~MB-49877~~).

Chris Hillery made changes - 03/Dec/21 2:26 AM

Assignee

Couchbase Build Team [ build-team ]

Chris Hillery [ ceej ]

Chris Hillery made changes - 03/Dec/21 2:26 AM

Sprint

Build Team 2021 Sprint 24 [ 1875 ]

Dave Rigby made changes - 03/Dec/21 2:40 AM

Description

When trying to debug a memcached issue on CB Server 7.1.0-1817 on CentOS 7, GDB is reporting that debugging information even when the -debuginfo RPM is installed:

{code}
rpm -qa|grep couchbase
couchbase-server-debuginfo-7.1.0-1817.x86_64
couchbase-server-7.1.0-1817.x86_64
{code}

h3. Observed Behaviour

GDB cannot find debugging information for our version of the C++ standard library {{/opt/couchbase/bin/../lib/libstdc++.so.6}}:

{code:highlight=21}
gdb -p $(pgrep memcached)
...
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f0d19a102c0 0x00007f0d19a1493e Yes /opt/couchbase/bin/../lib/libnuma.so.1
0x00007f0d19809fe0 0x00007f0d1980abfa Yes /opt/couchbase/bin/../lib/liburing.so.1
0x00007f0d195ee290 0x00007f0d19601670 Yes (*) /opt/couchbase/bin/../lib/libz.so.1
0x00007f0d193d0030 0x00007f0d193e2455 Yes /opt/couchbase/bin/../lib/libphosphor.so
0x00007f0d191c3e50 0x00007f0d191c494e Yes /lib64/libdl.so.2
0x00007f0d18fac8f0 0x00007f0d18fb7db1 Yes /lib64/libpthread.so.0
0x00007f0d18da1230 0x00007f0d18da41bc Yes /lib64/librt.so.1
0x00007f0d18b71b90 0x00007f0d18b9b3a7 Yes (*) /opt/couchbase/bin/../lib/liblz4.so.1
0x00007f0d189675c0 0x00007f0d1896c97f Yes /opt/couchbase/bin/../lib/libsnappy.so.1
0x00007f0d18495750 0x00007f0d1852abfa Yes /opt/couchbase/bin/../lib/libjemalloc.so.2
0x00007f0d1826a2f0 0x00007f0d182836c1 Yes /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
0x00007f0d18048e90 0x00007f0d1805961f Yes /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7
0x00007f0d17e3fd50 0x00007f0d17e400f7 Yes /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7
0x00007f0d17c3a6f0 0x00007f0d17c3caa4 Yes /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7
0x00007f0d179c20d0 0x00007f0d17a0fb16 Yes (*) /opt/couchbase/bin/../lib/libssl.so.1.1
0x00007f0d1751d000 0x00007f0d176dbf4a Yes (*) /opt/couchbase/bin/../lib/libcrypto.so.1.1
0x00007f0d17171ef0 0x00007f0d172498ba Yes (*) /opt/couchbase/bin/../lib/libstdc++.so.6
0x00007f0d16ddc350 0x00007f0d16e47316 Yes /lib64/libm.so.6
0x00007f0d16bc1af0 0x00007f0d16bd22dd Yes (*) /opt/couchbase/bin/../lib/libgcc_s.so.1
0x00007f0d168109f0 0x00007f0d1696093f Yes /lib64/libc.so.6
0x00007f0d19c18af0 0x00007f0d19c34050 Yes /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.
{code}

h3. Expected behaviour

Like our other libraries (e.g. /opt/couchbase/bin/../lib/libjemalloc.so.2) debugging information should be present.

h3. Initial analysis

The expected .debug file is missing from the RPM - for example here are all "lib" prefixed debug files in the lib directory - note no libstdc++:

{code}
# rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep /usr/lib/debug/opt/couchbase/lib/lib
/usr/lib/debug/opt/couchbase/lib/libbreakpad_wrapper.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libchrome_zlib.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchbase.so.8.0.5-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchstore.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchstore_views.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcurl.so.4.7.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libforestdb.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libicui18n.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libicuuc.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libjseval.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcre.so.1.2.12-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcrecpp.so.0.0.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcreposix.so.0.0.7-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libphosphor.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libsigar.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/liburing.so.1.0.6-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libuv.so.1.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8_libbase.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8_libplatform.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11-7.1.0-1817.x86_64.debug
{code}

(Seen while investigating ~~MB-49877~~).

When trying to debug a memcached issue on CB Server 7.1.0-1817 on CentOS 7, GDB is reporting that debugging information is missing even when the -debuginfo RPM is installed:

{code}
rpm -qa|grep couchbase
couchbase-server-debuginfo-7.1.0-1817.x86_64
couchbase-server-7.1.0-1817.x86_64
{code}

h3. Observed Behaviour

GDB cannot find debugging information for our version of the C++ standard library {{/opt/couchbase/bin/../lib/libstdc++.so.6}}:

{code:highlight=21}
gdb -p $(pgrep memcached)
...
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f0d19a102c0 0x00007f0d19a1493e Yes /opt/couchbase/bin/../lib/libnuma.so.1
0x00007f0d19809fe0 0x00007f0d1980abfa Yes /opt/couchbase/bin/../lib/liburing.so.1
0x00007f0d195ee290 0x00007f0d19601670 Yes (*) /opt/couchbase/bin/../lib/libz.so.1
0x00007f0d193d0030 0x00007f0d193e2455 Yes /opt/couchbase/bin/../lib/libphosphor.so
0x00007f0d191c3e50 0x00007f0d191c494e Yes /lib64/libdl.so.2
0x00007f0d18fac8f0 0x00007f0d18fb7db1 Yes /lib64/libpthread.so.0
0x00007f0d18da1230 0x00007f0d18da41bc Yes /lib64/librt.so.1
0x00007f0d18b71b90 0x00007f0d18b9b3a7 Yes (*) /opt/couchbase/bin/../lib/liblz4.so.1
0x00007f0d189675c0 0x00007f0d1896c97f Yes /opt/couchbase/bin/../lib/libsnappy.so.1
0x00007f0d18495750 0x00007f0d1852abfa Yes /opt/couchbase/bin/../lib/libjemalloc.so.2
0x00007f0d1826a2f0 0x00007f0d182836c1 Yes /opt/couchbase/bin/../lib/libevent_core-2.1.so.7
0x00007f0d18048e90 0x00007f0d1805961f Yes /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7
0x00007f0d17e3fd50 0x00007f0d17e400f7 Yes /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7
0x00007f0d17c3a6f0 0x00007f0d17c3caa4 Yes /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7
0x00007f0d179c20d0 0x00007f0d17a0fb16 Yes (*) /opt/couchbase/bin/../lib/libssl.so.1.1
0x00007f0d1751d000 0x00007f0d176dbf4a Yes (*) /opt/couchbase/bin/../lib/libcrypto.so.1.1
0x00007f0d17171ef0 0x00007f0d172498ba Yes (*) /opt/couchbase/bin/../lib/libstdc++.so.6
0x00007f0d16ddc350 0x00007f0d16e47316 Yes /lib64/libm.so.6
0x00007f0d16bc1af0 0x00007f0d16bd22dd Yes (*) /opt/couchbase/bin/../lib/libgcc_s.so.1
0x00007f0d168109f0 0x00007f0d1696093f Yes /lib64/libc.so.6
0x00007f0d19c18af0 0x00007f0d19c34050 Yes /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.
{code}

h3. Expected behaviour

Like our other libraries (e.g. /opt/couchbase/bin/../lib/libjemalloc.so.2) debugging information should be present.

h3. Initial analysis

The expected .debug file is missing from the RPM - for example here are all "lib" prefixed debug files in the lib directory - note no libstdc++:

{code}
# rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep /usr/lib/debug/opt/couchbase/lib/lib
/usr/lib/debug/opt/couchbase/lib/libbreakpad_wrapper.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libchrome_zlib.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchbase.so.8.0.5-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchstore.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcouchstore_views.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libcurl.so.4.7.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libforestdb.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libicui18n.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libicuuc.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libjseval.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcre.so.1.2.12-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcrecpp.so.0.0.2-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libpcreposix.so.0.0.7-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libphosphor.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libsigar.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.1.8-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libssl.so.1.1-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/liburing.so.1.0.6-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libuv.so.1.0.0-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8_libbase.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libv8_libplatform.so-7.1.0-1817.x86_64.debug
/usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11-7.1.0-1817.x86_64.debug
{code}

(Seen while investigating ~~MB-49877~~).

Chris Hillery added a comment - 03/Dec/21 3:07 AM

So libstdc++.so.6.0.29 that we install in /opt/couchbase/lib is not stripped, and we tweak the gcc build process to ensure it still has debug information in it: https://github.com/couchbase/build-infra/blob/master/docker/compile/gcc/centos-7.4/Dockerfile#L23-L29

I do see the following output from rpmbuild:

00:14:09.016 extracting debug info from /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server-0.0.0-2549.x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29

00:14:09.029 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server-0.0.0-2549.x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29: Unknown debugging section .debug_line_str

00:14:09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server-0.0.0-2549.x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29: Unknown debugging section .debug_loclists

00:14:09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server-0.0.0-2549.x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29: Unknown debugging section .debug_rnglists

00:14:09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server-0.0.0-2549.x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29: DWARF version 5 unhandled

Does that mean anything to you, Dave Rigby ?

Chris Hillery added a comment - 03/Dec/21 3:07 AM So libstdc++.so.6.0.29 that we install in /opt/couchbase/lib is not stripped, and we tweak the gcc build process to ensure it still has debug information in it: https://github.com/couchbase/build-infra/blob/master/docker/compile/gcc/centos-7.4/Dockerfile#L23-L29 I do see the following output from rpmbuild: 00 : 14 : 09.016 extracting debug info from /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server- 0.0 . 0 - 2549 .x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so. 6.0 . 29 00 : 14 : 09.029 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server- 0.0 . 0 - 2549 .x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so. 6.0 . 29 : Unknown debugging section .debug_line_str 00 : 14 : 09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server- 0.0 . 0 - 2549 .x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so. 6.0 . 29 : Unknown debugging section .debug_loclists 00 : 14 : 09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server- 0.0 . 0 - 2549 .x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so. 6.0 . 29 : Unknown debugging section .debug_rnglists 00 : 14 : 09.030 /usr/lib/rpm/debugedit: /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/voltron/staging/rpmbuild/BUILDROOT/couchbase-server- 0.0 . 0 - 2549 .x86_64/opt/couchbase/lib/python/interp/lib/libstdc++.so. 6.0 . 29 : DWARF version 5 unhandled Does that mean anything to you, Dave Rigby ?

Dave Rigby added a comment - 03/Dec/21 3:19 AM

Those seem related to a different instance of libstdc++ as used by python - which actually I do see in the debuginfo RPM:

# rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep libstdc++

/usr/lib/debug/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29-7.1.0-1817.x86_64.debug

Note however that's a different version of the our "main" one - 6.0.28 vs 6.0.29:

# rpm -ql couchbase-server-7.1.0-1817.x86_64|grep libstdc++.so

/opt/couchbase/lib/libstdc++.so

/opt/couchbase/lib/libstdc++.so.6

/opt/couchbase/lib/libstdc++.so.6.0.28

/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py

/opt/couchbase/lib/python/interp/lib/libstdc++.so

/opt/couchbase/lib/python/interp/lib/libstdc++.so.6

/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29

What I believe should happen is we indeed have an unstripped libstdc++.so binary created in the GCC build; then when that is packaged up via rpm it extracts the debug symbols and creates a .debug file (in the -debugfinfo RPM) - essentially the same behaviour as for our own binaries (e.g. memcached) or even other 3rd party libs like jemalloc - which for example both do have .debug files in the debuginfo RPM:

 rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep "memcached\|jemalloc"

/usr/lib/debug/opt/couchbase/bin/memcached-7.1.0-1817.x86_64.debug

/usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug

Dave Rigby added a comment - 03/Dec/21 3:19 AM Those seem related to a different instance of libstdc++ as used by python - which actually I do see in the debuginfo RPM: # rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep libstdc++ /usr/lib/debug/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29-7.1.0-1817.x86_64.debug Note however that's a different version of the our "main" one - 6.0.28 vs 6.0.29: # rpm -ql couchbase-server-7.1.0-1817.x86_64|grep libstdc++.so /opt/couchbase/lib/libstdc++.so /opt/couchbase/lib/libstdc++.so.6 /opt/couchbase/lib/libstdc++.so.6.0.28 /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py /opt/couchbase/lib/python/interp/lib/libstdc++.so /opt/couchbase/lib/python/interp/lib/libstdc++.so.6 /opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29 What I believe should happen is we indeed have an unstripped libstdc++.so binary created in the GCC build; then when that is packaged up via rpm it extracts the debug symbols and creates a .debug file (in the -debugfinfo RPM) - essentially the same behaviour as for our own binaries (e.g. memcached) or even other 3rd party libs like jemalloc - which for example both do have .debug files in the debuginfo RPM: rpm -ql couchbase-server-debuginfo-7.1.0-1817.x86_64|grep "memcached\|jemalloc" /usr/lib/debug/opt/couchbase/bin/memcached-7.1.0-1817.x86_64.debug /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2-7.1.0-1817.x86_64.debug

Chris Hillery added a comment - 03/Dec/21 3:29 AM

Right, sorry. Wrong libstdc+. But still, the right one /opt/couchbase/lib/libstdc+.so.6.0.28 is unstripped, so I'm a bit at a loss why rpmbuild wouldn't create any debug symbols for it.

I guess maybe "unstripped" and "has debug information" are different? Maybe there is no debug information in there to begin with, perhaps due to the way gcc was originally compiled?

Chris Hillery added a comment - 03/Dec/21 3:29 AM Right, sorry. Wrong libstdc+ . But still, the right one /opt/couchbase/lib/libstdc +.so.6.0.28 is unstripped, so I'm a bit at a loss why rpmbuild wouldn't create any debug symbols for it. I guess maybe "unstripped" and "has debug information" are different? Maybe there is no debug information in there to begin with, perhaps due to the way gcc was originally compiled?

Dave Rigby added a comment - 03/Dec/21 4:25 AM - edited

I guess maybe "unstripped" and "has debug information" are different? Maybe there is no debug information in there to begin with, perhaps due to the way gcc was originally compiled?

That's certainly possible - if there's nothing to put into the .debug I could see RPM skipping that file.

I've had a quick look at a few older versions

Latest 7.0.3
Missing:

# rpm -qpl couchbase-server-enterprise-debuginfo-7.0.3-7017-centos7.x86_64.rpm |grep libstdc

<EOF>

7.0.0 GA
Missing

# rpm -qpl couchbase-server-enterprise-debuginfo-7.0.0-centos7.x86_64.rpm |grep libstd

<EOF>

6.6.3 GA
Present:

# rpm -qpl couchbase-server-enterprise-debuginfo-6.6.3-centos7.x86_64.rpm |grep libstdc

/usr/lib/debug/opt/couchbase/lib/libstdc++.so.6.debug

/usr/lib/debug/opt/couchbase/lib/libstdc++.so.debug

So seems to be something which changed between 6.6.3 and 7.0.0.

Dave Rigby added a comment - 03/Dec/21 4:25 AM - edited I guess maybe "unstripped" and "has debug information" are different? Maybe there is no debug information in there to begin with, perhaps due to the way gcc was originally compiled? That's certainly possible - if there's nothing to put into the .debug I could see RPM skipping that file. I've had a quick look at a few older versions Latest 7.0.3 Missing: # rpm -qpl couchbase-server-enterprise-debuginfo-7.0.3-7017-centos7.x86_64.rpm |grep libstdc <EOF> 7.0.0 GA Missing # rpm -qpl couchbase-server-enterprise-debuginfo-7.0.0-centos7.x86_64.rpm |grep libstd <EOF> 6.6.3 GA Present: # rpm -qpl couchbase-server-enterprise-debuginfo-6.6.3-centos7.x86_64.rpm |grep libstdc /usr/lib/debug/opt/couchbase/lib/libstdc++.so.6.debug /usr/lib/debug/opt/couchbase/lib/libstdc++.so.debug So seems to be something which changed between 6.6.3 and 7.0.0.

Dave Rigby made changes - 03/Dec/21 4:25 AM

Affects Version/s

7.0.0 [ 17233 ]

Dave Rigby added a comment - 03/Dec/21 4:31 AM

I guess maybe "unstripped" and "has debug information" are different?

Just to clarify my previous response - these are indeed different. "unstripped" just means that no information has been removed from the original binary. However, exactly what information was in the original binary is another matter - i.e. it may well not have been compiled with DWARF debugging information (-g flag to compiler).

Dave Rigby added a comment - 03/Dec/21 4:31 AM I guess maybe "unstripped" and "has debug information" are different? Just to clarify my previous response - these are indeed different. "unstripped" just means that no information has been removed from the original binary. However, exactly what information was in the original binary is another matter - i.e. it may well not have been compiled with DWARF debugging information ( -g flag to compiler).

Chris Hillery added a comment - 03/Dec/21 4:43 AM

What changed between 6.6.3 and 7.0.0 was switching to gcc 10.

So that would require re-compiling gcc and figuring out how to embed debug information. Do you think that's safe at this point in the cycle? Couldn't that have implications beyond just having debug symbols for libstdc++, like possibly performance issues?

Chris Hillery added a comment - 03/Dec/21 4:43 AM What changed between 6.6.3 and 7.0.0 was switching to gcc 10. So that would require re-compiling gcc and figuring out how to embed debug information. Do you think that's safe at this point in the cycle? Couldn't that have implications beyond just having debug symbols for libstdc++, like possibly performance issues?

Dave Rigby added a comment - 03/Dec/21 4:49 AM

{{-g]} Doesn't change the actual generated code which is run - it just also generates the DWARF debug info; so it would have zero impact on the performance of libstdc++.so.

I think it's safe to do - IIRC for GCC 7 we just added a standard flag somewhere in it's configure steps; I would expect gcc 10 is similar.

Dave Rigby added a comment - 03/Dec/21 4:49 AM {{-g]} Doesn't change the actual generated code which is run - it just also generates the DWARF debug info; so it would have zero impact on the performance of libstdc++.so. I think it's safe to do - IIRC for GCC 7 we just added a standard flag somewhere in it's configure steps; I would expect gcc 10 is similar.

Chris Hillery made changes - 05/Dec/21 11:18 PM

Remote Link

This issue links to "Page (Couchbase, Inc. Wiki)" [ 23538 ]

Chris Hillery made changes - 05/Dec/21 11:18 PM

Status

Open [ 1 ]

In Progress [ 3 ]

Chris Hillery made changes - 06/Dec/21 2:02 PM

Sprint

Build Team 2021 Sprint 24 [ 1875 ]

Build Team 2021 Sprint 24, Build Team 2021 Sprint 25 [ 1875, 1894 ]

Wayne Siu made changes - 04/Jan/22 6:44 PM

Sprint

Build Team 2021 Sprint 24, Build Team 2021 Sprint 25 [ 1875, 1894 ]

Build Team 2021 Sprint 24, Build Team 2021 Sprint 25, Build Team 2022 Sprint 1 [ 1875, 1894, 1955 ]

Chris Hillery made changes - 17/Jan/22 2:48 PM

Sprint

Build Team 2021 Sprint 24, Build Team 2021 Sprint 25, Build Team 2022 Sprint 1 [ 1875, 1894, 1955 ]

Build Team 2021 Sprint 24, Build Team 2021 Sprint 25, Build Team 2022 Sprint 1, Build Team 2022 Sprint 2 [ 1875, 1894, 1955, 1956 ]

Chris Hillery added a comment - 20/Jan/22 12:42 AM

So... The libstdc+.so.6.0.28 file we put into /opt/couchbase does have debug information in it, as verified with various readelf commands. It does have a .debug_info section. I also tried rebuilding all of gcc with CXXFLAGS='-g3 -O0 -fno-inline' as recommended by the doc to get libstdc+ with debug symbols (https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html), and the resulting libstdc++.so.6.0.28 file was exactly the same size.

I know next to nothing about how the rpm/dpkg processes strip debug information out to a separate -debug package, so any pointers at where I could look next would be welcome. I'll Google around a bit.

Chris Hillery added a comment - 20/Jan/22 12:42 AM So... The libstdc+ .so.6.0.28 file we put into /opt/couchbase does have debug information in it, as verified with various readelf commands. It does have a .debug_info section. I also tried rebuilding all of gcc with CXXFLAGS='-g3 -O0 -fno-inline' as recommended by the doc to get libstdc + with debug symbols ( https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html ), and the resulting libstdc++.so.6.0.28 file was exactly the same size. I know next to nothing about how the rpm/dpkg processes strip debug information out to a separate -debug package, so any pointers at where I could look next would be welcome. I'll Google around a bit.

Chris Hillery added a comment - 20/Jan/22 2:15 AM - edited

Ran a few tests suggested by Dave Rigby to confirm that the library is indeed built with debug info, and also confirmed that this bug is unique to rpm - the -dbg.deb files DO have it.

After digging down into what rpm actually does, I eventually discovered a cryptic find command in /usr/lib/rpm/find-debuginfo.sh which filters out files that have any execute bit set; those are the candidate files for extracting debug info. /opt/couchbase/lib/libstdc++.so.6.0.28 has permissions 644. Sigh. The fix is simple, at least - add PERMISSIONS to the INSTALL() so the resulting file in /opt/couchbase/lib has permissions 755.

Chris Hillery added a comment - 20/Jan/22 2:15 AM - edited Ran a few tests suggested by Dave Rigby to confirm that the library is indeed built with debug info, and also confirmed that this bug is unique to rpm - the -dbg.deb files DO have it. After digging down into what rpm actually does, I eventually discovered a cryptic find command in /usr/lib/rpm/find-debuginfo.sh which filters out files that have any execute bit set; those are the candidate files for extracting debug info. /opt/couchbase/lib/libstdc++.so.6.0.28 has permissions 644. Sigh. The fix is simple, at least - add PERMISSIONS to the INSTALL() so the resulting file in /opt/couchbase/lib has permissions 755.

Chris Hillery made changes - 20/Jan/22 2:36 AM

Remote Link

This issue links to "Page (Couchbase, Inc. Wiki)" [ 23622 ]

Chris Hillery added a comment - 20/Jan/22 2:37 AM

Confirmed with toy build that my fix works. Currently seeking approval to submit this change to the cheshire-cat branch as requested by Dave Rigby so it can be merged forward to master/Neo.

Chris Hillery added a comment - 20/Jan/22 2:37 AM Confirmed with toy build that my fix works. Currently seeking approval to submit this change to the cheshire-cat branch as requested by Dave Rigby so it can be merged forward to master/Neo.

Wayne Siu made changes - 20/Jan/22 11:53 PM

Link

This issue blocks MB-48997 [ MB-48997 ]

Chris Hillery made changes - 21/Jan/22 3:17 AM

Fix Version/s

6.6.5 [ 17614 ]

Chris Hillery made changes - 21/Jan/22 3:18 AM

Fix Version/s		CheshireCat.Next [ 16908 ]
Fix Version/s	6.6.5 [ 17614 ]

Chris Hillery added a comment - 21/Jan/22 3:20 AM

Permission granted; change is merged to cheshire-cat and master branches.

Chris Hillery added a comment - 21/Jan/22 3:20 AM Permission granted; change is merged to cheshire-cat and master branches.

Permalink

Chris Hillery logged work - 21/Jan/22 3:21 AM

Time Spent:

4h

<No comment>

Chris Hillery made changes - 21/Jan/22 3:21 AM

Assignee	Chris Hillery [ ceej ]	Dave Rigby [ drigby ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

Chris Hillery made changes - 21/Jan/22 3:21 AM

Worklog Id	15861 [ 15861 ]
Remaining Estimate		0h [ 0 ]
Time Spent		4h [ 14400 ]

Chris Hillery made changes - 21/Jan/22 3:25 AM

Fix Version/s

CheshireCat.Next [ 16908 ]

Chris Hillery made changes - 21/Jan/22 3:25 AM

Labels

candidate-for-7.0.4

Couchbase Build Team added a comment - 21/Jan/22 4:50 AM

Build couchbase-server-7.0.4-7211 contains tlm commit d403072 with commit message:
~~MB-49881~~: Make libstdc++ etc. executable to make rpm happy

Couchbase Build Team added a comment - 21/Jan/22 4:50 AM Build couchbase-server-7.0.4-7211 contains tlm commit d403072 with commit message: MB-49881 : Make libstdc++ etc. executable to make rpm happy

Couchbase Build Team added a comment - 21/Jan/22 4:52 AM

Build couchbase-server-7.1.0-2122 contains tlm commit d403072 with commit message:
~~MB-49881~~: Make libstdc++ etc. executable to make rpm happy

Couchbase Build Team added a comment - 21/Jan/22 4:52 AM Build couchbase-server-7.1.0-2122 contains tlm commit d403072 with commit message: MB-49881 : Make libstdc++ etc. executable to make rpm happy

Ritam Sharma made changes - 24/Jan/22 2:37 AM

Labels

candidate-for-7.0.4

candidate-for-7.0.4 request-dev-verify

Wayne Siu made changes - 10/Mar/22 11:08 PM

Labels

candidate-for-7.0.4 request-dev-verify

approved-for-7.0.3 candidate-for-7.0.4 request-dev-verify

Wayne Siu made changes - 10/Mar/22 11:09 PM

Labels

approved-for-7.0.3 candidate-for-7.0.4 request-dev-verify

approved-for-7.0.4 candidate-for-7.0.4 request-dev-verify

Wayne Siu made changes - 10/Mar/22 11:09 PM

Fix Version/s

7.0.4 [ 18322 ]

Dave Rigby made changes - 31/Mar/22 1:56 AM

VERIFICATION STEPS		Confirmed that debug RPM now includes debug symbols for libstdc++: {code} # rpm -qpl couchbase-server-enterprise-debuginfo-7.1.0-2535-centos7.x86_64.rpm \|grep libstdc\+\+ /usr/lib/debug/opt/couchbase/lib/libstdc++.so.6.0.28-7.1.0-2535.x86_64.debug /usr/lib/debug/opt/couchbase/lib/python/interp/lib/libstdc++.so.6.0.29-7.1.0-2535.x86_64.debug {code}
Status	Resolved [ 5 ]	Closed [ 6 ]

Couchbase Build Team added a comment - 04/Aug/22 12:29 PM

Build couchbase-lite-core-3.1.0-205 contains tlm commit d403072 with commit message:
~~MB-49881~~: Make libstdc++ etc. executable to make rpm happy

Couchbase Build Team added a comment - 04/Aug/22 12:29 PM Build couchbase-lite-core-3.1.0-205 contains tlm commit d403072 with commit message: MB-49881 : Make libstdc++ etc. executable to make rpm happy

People

Assignee:: Dave Rigby

Reporter:: Dave Rigby

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 03/Dec/21 1:51 AM

Updated:: 04/Aug/22 12:29 PM

Resolved:: 21/Jan/22 3:21 AM

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

Gerrit Reviews

There are no open Gerrit changes

Show There are 3 closed Gerrit changes

Hide There are 3 closed Gerrit changes

MB-49881: Make libstdc++ etc. executable to make rpm happy: Gerrit Review:

MB-49881: Make libstdc++ etc. executable to make rpm happy: Gerrit Review:

Merge branch 'cheshire-cat' into master: Gerrit Review:

PagerDuty

Couchbase Server

Details

Description

Observed Behaviour

Expected behaviour

Initial analysis

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Time Tracking

Gerrit Reviews

PagerDuty