Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53818

[Magma] - GSL: Precondition failure: 'compactions->find(lowStr) == compactions->end()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:1137

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Steps to Repro
      1. Create a 5 node cluster.

      2022-09-23 19:12:11,303 | test  | INFO    | MainThread | [table_view:display:72] Cluster statistics
      +----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
      | Nodes          | Zone    | Services | CPU    | Mem_total | Mem_free  | Swap_mem_used        | Active / Replica  | Version / Config                   |
      +----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
      | 172.23.108.251 | Group 1 | kv       | 0.3049 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB  | 0 / 0             | 7.2.0-2132-enterprise / serverless |
      | 172.23.105.36  | Group 1 | kv       | 0.4898 | 11.45 GiB | 10.69 GiB | 39.41 MiB / 3.50 GiB | 0 / 0             | 7.2.0-2132-enterprise / serverless |
      | 172.23.105.33  | Group 1 | kv       | 1.2384 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB  | 0 / 0             | 7.2.0-2132-enterprise / serverless |
      | 172.23.104.76  | Group 1 | kv       | 1.1269 | 11.45 GiB | 10.58 GiB | 0.0 Byte / 3.50 GiB  | 0 / 0             | 7.2.0-2132-enterprise / serverless |
      | 172.23.121.79  | Group 1 | kv       | 1.1837 | 11.45 GiB | 10.68 GiB | 0.0 Byte / 3.50 GiB  | 0 / 0             | 7.2.0-2132-enterprise / serverless |
      +----------------+---------+----------+--------+-----------+-----------+----------------------+-------------------+------------------------------------+
      

      2. Create 10 magma bucket of 256 MB and push these buckets to 1 DGM.
      3. Add a node(172.23.123.5) , remove a node(172.23.104.76) and start a swap rebalace.

      After swap rebalance of some 7 buckets successfully, I noticed following messages during 8th bucket rebalance and rebalance failed.

      172.23.123.5 10:34:33 AM 25 Sep, 2022

      Service 'memcached' exited with status 134. Restarting. Messages:
      2022-09-23T19:10:44.977099-07:00 WARNING Unknown key "enforce_tenant_limits_enabled" in config ignored.
      2022-09-25T10:34:32.914842-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2022-09-25T10:34:32.915100-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'compactions->find(lowStr) == compactions->end()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:1137
      terminate called after throwing an instance of 'gsl::fail_fast'
      what(): GSL: Precondition failure: 'compactions->find(lowStr) == compactions->end()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:1137
      

      172.23.104.76 10:34:33 AM 25 Sep, 2022

      Worker <0.4924.1993> (for action {move,
      {26,
      ['ns_1@172.23.105.36',
      'ns_1@172.23.104.76',
      'ns_1@172.23.105.33'],
      ['ns_1@172.23.105.36',
      'ns_1@172.23.123.5',
      'ns_1@172.23.105.33'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT',
      <0.31864.1992>,
      {{dcp_wait_for_data_move_failed,
      "default8",
      26,
      'ns_1@172.23.105.36',
      ['ns_1@172.23.123.5',
      'ns_1@172.23.105.33'],
      {error,
      no_stats_for_this_vbucket}},
      [{ns_single_vbucket_mover,
      '-wait_dcp_data_move/5-fun-0-',
      5,
      [{file,
      "src/ns_single_vbucket_mover.erl"},
      {line,
      451}]},
      {proc_lib,
      init_p,3,
      [{file,
      "proc_lib.erl"},
      {line,
      211}]}]}}}
      

      172.23.104.76 10:34:33 AM 25 Sep, 2022

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.31864.1992>,
      {{dcp_wait_for_data_move_failed,"default8",
      26,'ns_1@172.23.105.36',
      ['ns_1@172.23.123.5','ns_1@172.23.105.33'],
      {error,no_stats_for_this_vbucket}},
      [{ns_single_vbucket_mover,
      '-wait_dcp_data_move/5-fun-0-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},
      {line,451}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,211}]}]}}}}.
      Rebalance Operation Id = 4b6a2acdcb56b546137190843cf52cf7
      

      cbcollect_info attached.

      Attachments

        For Gerrit Dashboard: MB-53818
        # Subject Branch Project Status CR V

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty