Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
1006.5.1-1041
-
Untriaged
-
-
Unknown
Description
1. 3 nodes in the cluster, 1 bucket, 1M items
2. Rebalance in 1 more node and start data loading at the same time.
3. Kick in full compaction for the bucket.
Observe the WAL corruption messages on the logs and crash dumps created.
On Node .16:
2020-01-31T05:15:41.343585-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2020-01-31T05:15:41.343642-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): InsertTables: Tree Snapshot:
|
TreeSnapshot :
|
------------------------------------------------------
|
BloomFilterSize:0, IndexBlocksSize:1512
|
------------------------------------------------------
|
# Level:0, Size:81920, LogicalSize:15417 #
|
ID:18 Range:(k(8)=0x0000000000000001(' '), sn=1656, op=delta, k(8)=0x0000000000000250(' P'), sn=1816, op=delta) Size:20480 LogicalSize:5590 AvgDataBlockSize:761 CompressionRatio:2.0749
|
ID:21 Range:(k(8)=0x000000000000004a(' J'), sn=1820, op=delta, k(8)=0x0000000000000262(' b'), sn=1846, op=delta) Size:20480 LogicalSize:1138 AvgDataBlockSize:191 CompressionRatio:1.73822
|
ID:23 Range:(k(8)=0x0000000000000066(' f'), sn=1871, op=delta, k(8)=0x00000000000002b2(' '), sn=2063, op=delta) Size:20480 LogicalSize:7863 AvgDataBlockSize:1062 CompressionRatio:2.08757
|
ID:26 Range:(k(8)=0x00000000000000c1(' '), sn=2073, op=delta, k(8)=0x00000000000002b9(' '), sn=2087, op=delta) Size:20480 LogicalSize:826 AvgDataBlockSize:140 CompressionRatio:1.75
|
------------------------------------------------------
|
# Level:1, Size:0, LogicalSize:0 #
|
------------------------------------------------------
|
# Level:2, Size:0, LogicalSize:0 #
|
------------------------------------------------------
|
# Level:3, Size:0, LogicalSize:0 #
|
------------------------------------------------------
|
# Level:4, Size:225280, LogicalSize:185846 #
|
ID:2 Range:(k(8)=0x0000000000000001(' '), sn=7, op=update, k(8)=0x00000000000000d0(' '), sn=351, op=update) Size:20480 LogicalSize:34431 AvgDataBlockSize:1123.67 CompressionRatio:3.39652
|
ID:4 Range:(k(8)=0x00000000000000d1(' '), sn=358, op=update, k(8)=0x000000000000019c(' '), sn=692, op=update) Size:20480 LogicalSize:33753 AvgDataBlockSize:1088 CompressionRatio:3.44363
|
ID:6 Range:(k(8)=0x000000000000019d(' '), sn=701, op=update, k(8)=0x0000000000000237(' 7'), sn=960, op=update) Size:20480 LogicalSize:25654 AvgDataBlockSize:1046.71 CompressionRatio:3.49679
|
ID:8 Range:(k(8)=0x0000000000000238(' 8'), sn=968, op=update, k(8)=0x00000000000002cb(' '), sn=1203, op=update) Size:20480 LogicalSize:24485 AvgDataBlockSize:1177.17 CompressionRatio:3.4576
|
ID:10 Range:(k(8)=0x00000000000002cc(' '), sn=1210, op=update, k(8)=0x000000000000034b(' K'), sn=1433, op=update) Size:20480 LogicalSize:21185 AvgDataBlockSize:1043.67 CompressionRatio:3.38215
|
ID:12 Range:(k(8)=0x000000000000034c(' L'), sn=1438, op=update, k(8)=0x00000000000003c0(' '), sn=1628, op=update) Size:20480 LogicalSize:19364 AvgDataBlockSize:1132.8 CompressionRatio:3.41049
|
ID:14 Range:(k(8)=0x00000000000003c1(' '), sn=1633, op=update, k(8)=0x00000000000003ce(' '), sn=1651, op=update) Size:20480 LogicalSize:2319 AvgDataBlockSize:747 CompressionRatio:3.10442
|
ID:19 Range:(k(8)=0x00000000000003cf(' '), sn=1655, op=update, k(8)=0x0000000000000404(' '), sn=1813, op=update) Size:20480 LogicalSize:8938 AvgDataBlockSize:866 CompressionRatio:3.45304
|
ID:22 Range:(k(8)=0x0000000000000405(' '), sn=1821, op=update, k(8)=0x000000000000040f(' '), sn=1847, op=update) Size:20480 LogicalSize:1820 AvgDataBlockSize:634 CompressionRatio:2.87539
|
ID:24 Range:(k(8)=0x0000000000000410(' '), sn=1852, op=update, k(8)=0x000000000000045b(' ['), sn=2066, op=update) Size:20480 LogicalSize:12575 AvgDataBlockSize:918.5 CompressionRatio:3.41726
|
ID:27 Range:(k(8)=0x000000000000045c(' \'), sn=2070, op=update, k(8)=0x0000000000000463(' c'), sn=2082, op=update) Size:20480 LogicalSize:1322 AvgDataBlockSize:478 CompressionRatio:2.77824
|
------------------------------------------------------
|
Level:4 Table List : | ID:29 Range:(k(8)=0x0000000000000456(' V'), sn=2091, op=update, k(8)=0x000000000000045b(' ['), sn=2100, op=update) Size:20480 LogicalSize:992 AvgDataBlockSize:379 CompressionRatio:2.63852 |
|
|
|
|
|
2020-01-31T05:15:41.428566-08:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1041). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/22d79707-595e-a0e9-5fb41fcd-6a4ce65e.dmp before terminating.
|
2020-01-31T05:15:41.428586-08:00 CRITICAL Stack backtrace of crashed thread:
|
2020-01-31T05:15:41.428732-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
|
2020-01-31T05:15:41.428744-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
|
2020-01-31T05:15:41.428753-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
|
2020-01-31T05:15:41.428760-08:00 CRITICAL /lib64/libpthread.so.0() [0x7fc7d4bbc000+0xf5d0]
|
2020-01-31T05:15:41.428784-08:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7fc7d47ef000+0x36207]
|
2020-01-31T05:15:41.428819-08:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fc7d47ef000+0x378f8]
|
2020-01-31T05:15:41.428856-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fc7d52f1000+0x91195]
|
2020-01-31T05:15:41.428865-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x145e02]
|
2020-01-31T05:15:41.428881-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc7d52f1000+0x8ef86]
|
2020-01-31T05:15:41.428896-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc7d52f1000+0x8efd1]
|
2020-01-31T05:15:41.428910-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc7d52f1000+0x8f213]
|
2020-01-31T05:15:41.428917-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0x5fd09]
|
2020-01-31T05:15:41.428923-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0xc9961]
|
2020-01-31T05:15:41.428928-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0xb524d]
|
2020-01-31T05:15:41.428933-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0xbd8c7]
|
2020-01-31T05:15:41.428936-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0xbf098]
|
2020-01-31T05:15:41.428941-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fc7cf70b000+0x5398f]
|
2020-01-31T05:15:41.428946-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fc7d719a000+0x8f17]
|
2020-01-31T05:15:41.428952-08:00 CRITICAL /lib64/libpthread.so.0() [0x7fc7d4bbc000+0x7dd5]
|
2020-01-31T05:15:41.428982-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fc7d47ef000+0xfdead]
|
2020-01-31T05:15:41.430385-08:00 INFO ---------- Closing logfile
|
On Node .17:
memcached.log.001075.txt:2020-01-31T08:37:03.771137-08:00 CRITICAL WALIterator corruption error:Corruption: 9:60006400
|
memcached.log.001075.txt:2020-01-31T08:37:10.155086-08:00 CRITICAL WALIterator corruption error:Corruption: 9:25395200
|
memcached.log.001075.txt:2020-01-31T08:37:13.316726-08:00 CRITICAL WAL replay at offset 8:58830848 must start with a BeginTx
|
memcached.log.001075.txt:2020-01-31T08:37:13.318753-08:00 CRITICAL (default) magma_2 MagmaKVStore Magma open failed. Status:Corruption: Failed to initialize wal iterator :NotFound: WAL replay at offset 8:58830848 must start with a BeginTx
|
memcached.log.001075.txt:2020-01-31T08:37:13.318902-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
memcached.log.001075.txt:2020-01-31T08:37:13.318955-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): MagmaKVStore Magma open failed. Status:Corruption: Failed to initialize wal iterator :NotFound: WAL replay at offset 8:58830848 must start with a BeginTx
|
memcached.log.001075.txt:2020-01-31T08:37:13.410522-08:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1041). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/0cdcc8c2-950b-1893-2b223991-687e4433.dmp before terminating.
|
memcached.log.001075.txt:2020-01-31T08:37:13.410565-08:00 CRITICAL Stack backtrace of crashed thread:
|
memcached.log.001075.txt:2020-01-31T08:37:13.410768-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410789-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410798-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410808-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f2fcc084000+0xf5d0]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410834-08:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f2fcbcb7000+0x36207]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410861-08:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f2fcbcb7000+0x378f8]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410891-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f2fcc7b9000+0x91195]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410902-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x145e02]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410925-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fcc7b9000+0x8ef86]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410941-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fcc7b9000+0x8efd1]
|
memcached.log.001075.txt:2020-01-31T08:37:13.410955-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fcc7b9000+0x8f213]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411026-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x1fbcbc]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411037-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x1466a7]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411046-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x160b54]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411053-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x1a3f93]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411087-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x1518ae]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411101-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0xd9596]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411108-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0xef189]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411113-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f2fc728a000+0x10254f]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411121-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x24118]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411129-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x245d9]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411135-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7f2fce662000+0x147bf]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411140-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f2fce662000+0x8f17]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411146-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f2fcc084000+0x7dd5]
|
memcached.log.001075.txt:2020-01-31T08:37:13.411177-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f2fcbcb7000+0xfdead]
|