Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45395

Error messages like "CouchKVStore::compactDB: exception while performing compaction for vb:241 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file" seen in memcached logs

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes
    • KV-Engine 2021-March

    Description

      Script to Repro

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.4521.ini GROUP=rebalance_with_collection_crud_dgm,rerun=False,upgrade_version=7.0.0-4834 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=dgm.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,dgm=55,num_items=32000,skip_validations=False,GROUP=rebalance_with_collection_crud_dgm'
      

      Steps to Repro
      1) Create a 5 node cluster.
      2021-04-01 01:05:14,032 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview
      -----------------------------------------------------------------------

      Nodes Services Version CPU Status

      -----------------------------------------------------------------------

      172.23.106.13 kv 7.0.0-4834-enterprise 0.376695128076 Cluster node
      172.23.106.16 None     <--- IN —
      172.23.106.15 None     <--- IN —
      172.23.106.11 None     <--- IN —
      172.23.104.251 None     <--- IN —

      -----------------------------------------------------------------------

      2) Create bucket/scope/collections.data
      2021-04-01 01:07:43,092 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
      ----------------------------------------------------------------------------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      ----------------------------------------------------------------------------

      default couchbase 2 none 0 6400000 4194304000 2390737096 4158093834

      ----------------------------------------------------------------------------

      3) Push the bucket to DGM

      2021-04-01 01:07:51,966 | test  | INFO    | pool-3-thread-4 | [task:_load_bucket_into_dgm:2082] DGM doc loading for 'default' to atleast 55%
      

      4) Rebalance out 2 nodes (172.23.106.11 and 172.23.104.251)
      2021-04-01 01:07:55,461 | test | INFO | pool-3-thread-5 | [table_view:display:72] Rebalance Overview
      ----------------------------------------------------------------------

      Nodes Services Version CPU Status

      ----------------------------------------------------------------------

      172.23.106.16 kv 7.0.0-4834-enterprise 9.55882352941 Cluster node
      172.23.106.13 kv 7.0.0-4834-enterprise 9.50695322377 Cluster node
      172.23.106.15 kv 7.0.0-4834-enterprise 6.45079686314 Cluster node
      172.23.106.11 kv 7.0.0-4834-enterprise 5.90759909114 — OUT --->
      172.23.104.251 kv 7.0.0-4834-enterprise 6.08040201005 — OUT --->

      ----------------------------------------------------------------------

      Following error messages are seen on 172.23.106.15

      [basetestcase:check_coredump_exist:795] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt
      2021-04-01 01:09:46,092 | test  | CRITICAL | MainThread | [basetestcase:check_coredump_exist:797] 172.23.106.15: Found ' ERROR ' logs - ['2021-04-01T01:08:00.924756-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:241 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n', '2021-04-01T01:08:01.002835-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:245 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n']
      

      cbcollect_info attached. This was not seen on 7.0.0-4797.

      Attachments

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty