Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44694

[Collections] - ERROR messages related to CouchKVStore::compactDBInternal seen in memcached logs

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Script to Repro

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.14263.ini GROUP=durability_persist_to_majority,rerun=False,upgrade_version=7.0.0-4578 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority'
      

      Steps to Repro
      1) Create a 4 node cluster
      2021-03-02 04:45:51,967 | test | INFO | pool-4-thread-7 | [table_view:display:72] Rebalance Overview
      ----------------------------------------------------------------------

      Nodes Services Version CPU Status

      ----------------------------------------------------------------------

      172.23.121.214 kv 7.0.0-4578-enterprise 1.97648236177 Cluster node
      172.23.123.99 None     <--- IN —
      172.23.123.96 None     <--- IN —
      172.23.123.97 None     <--- IN —

      ----------------------------------------------------------------------

      2) Create buckets/scopes/collections/data.
      2021-03-02 04:51:02,661 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
      -------------------------------------------------------------------------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      -------------------------------------------------------------------------

      bucket1 couchbase 2 none 0 30000 419430400 112321568 307977480
      bucket2 couchbase 2 none 0 30000 419430400 112827184 194371822
      default couchbase 2 none 0 500000 6291456000 417488272 419289625

      -------------------------------------------------------------------------

      4)Add 172.23.123.98 and 172.23.121.213 , remove(172.23.123.97 and 172.23.123.96) and start a swap rebalance of 2 nodes.
      2021-03-02 04:51:13,318 | test | INFO | pool-4-thread-1 | [table_view:display:72] Rebalance Overview
      -----------------------------------------------------------------------

      Nodes Services Version CPU Status

      -----------------------------------------------------------------------

      172.23.123.98 kv 7.0.0-4578-enterprise 0.326387145368 Cluster node
      172.23.123.97 kv 7.0.0-4578-enterprise 7.98383021728 — OUT --->
      172.23.121.213 kv 7.0.0-4578-enterprise 0.551240290654 Cluster node
      172.23.123.96 kv 7.0.0-4578-enterprise 7.07427993936 — OUT --->
      172.23.123.99 kv 7.0.0-4578-enterprise 6.18191031163 Cluster node
      172.23.121.214 kv 7.0.0-4578-enterprise 8.39195979899 Cluster node

      -----------------------------------------------------------------------

      We see ERROR messages like below on 172.23.121.213

      2021-03-02 04:55:28,926 | test  | INFO    | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt
      2021-03-02 04:55:28,927 | test  | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.121.213: Found ' ERROR ' logs - ['2021-03-02T04:55:17.490262-08:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:118 - Details: CouchKVStore::compactDBInternal finalising dropped collections, container should be emptyEraserContext: removed:true, ScanContext: startSeqno:0, endSeqno:1416 dropped:[0x62, 0x61, 0x63, 0x64, 0x65, 0x5e, 0x5d, 0x5f, 0x60, 0x58, 0x57, 0x59, 0x5a, 0x5b, 0x5c, 0x55, 0x54, 0x23, 0x21, 0x22, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x1c, 0x1b, 0x1a, 0x19, 0x17, 0x18, 0x1d, 0x1e, 0x1f, 0x20, 0xb, 0xa, 0x9, 0x4f, 0x11, 0xd, 0x2d, 0x8, 0xc, 0x12, 0x10, 0xe, 0xf, 0x16, 0x15, 0x13, 0x14, 0x24, 0x25, 0x26, 0x32, 0x31, 0x30, 0x2e, 0x2f, 0x37, 0x36, 0x35, 0x33, 0x34, 0x3b, 0x3a, 0x38, 0x39, 0x41, 0x40, 0x3f, 0x3e, 0x3c, 0x3d, 0x45, 0x44, 0x42, 0x43, 0x4b, 0x4a, 0x49, 0x48, 0x46, 0x47, 0x50, 0x4e, 0x4c, 0x4d, 0x53, 0x51, 0x52, 0x56, ]\n']
      

      cbcollect_info attached. This was not seen on 7.0.0-4554.

      Attachments

        For Gerrit Dashboard: MB-44694
        # Subject Branch Project Status CR V

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty