memcached crashes in Collections::VB::ManifestEntry::getItemCount

Steps:

1. Cluster setup

   +---------------+-------------+--------+-------------------+

   | Nodes         | Services    | CPU    | Active / Replica  |

   +---------------+-------------+--------+-------------------+

   | 172.23.108.70 | index       | 7.7144 | 0 / 0             |

   | 172.23.108.69 | kv          | 79.541 | 225124 / 225333   |

   | 172.23.108.74 | index, n1ql | 3.2889 | 0 / 0             |

   | 172.23.108.67 | kv          | 78.250 | 214981 / 214182   |

   | 172.23.108.72 | index       | 4.1314 | 0 / 0             |

   | 172.23.108.68 | kv          | 78.619 | 225834 / 225982   |

   +---------------+-------------+--------+-------------------+

1. Bucket setup

   +---------+-----------+------------+----------+--------+-----------+------------+------------+-----+

   | Bucket  | Type      | Storage    | Replicas | Items  | RAM Quota | RAM Used   | Disk Used  | ARR |

   +---------+-----------+------------+----------+--------+-----------+------------+------------+-----+

   | bucket1 | couchbase | couchstore | 1        | 125000 | 1.50 GiB  | 95.55 MiB  | 178.43 MiB | 100 |

   | bucket2 | ephemeral | -          | 1        | 125000 | 3.00 GiB  | 116.51 MiB | 0.0 Byte   | -   |

   | default | couchbase | magma      | 1        | 500000 | 1.50 GiB  | 527.66 MiB | 984.27 MiB | 100 |

   +---------+-----------+------------+----------+--------+-----------+------------+------------+-----+

1. Setting oso_dcp_backfill=auto using diag_eval
2. Load data into all buckets
3. Create deferred index and wait for indexes to become available

Observation:

Memcached crashed exactly while creating indexes.

memcached.log.000028.txt: 2023-10-12T05:21:50.813151-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.0-1606). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/2e270040-57e0-4dab-4786d1b5-69f55a47.dmp before terminating. Writing dump succeeded: yes

GDB backtrace: (full_bt.txt)

#0  0x000000000086e524 in std::__atomic_base<unsigned long>::load (__m=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/atomic.h:503

[Current thread is 1 (LWP 48723)]

#0  0x000000000086e524 in std::__atomic_base<unsigned long>::load (__m=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/atomic.h:503

        __b = <optimized out>

        __b = <optimized out>

#1  cb::AtomicNonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::load (this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/bitset:235

No locals.

...

...