Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50956

Compaction started during rebalance got stuck

    XMLWordPrintable

Details

    Description

       

      Build: 7.1.0-2284

       

      Scenario:

      • 5 node cluster
      • 3 couchstore buckets (5 scopes per bucket and 40 collections / scope)

        +---------+-----------+-----------------+----------+------------+-----+--------+-------------+------------+------------+-----+
        | Bucket  | Type      | Storage Backend | Replicas | Durability | TTL | Items  | RAM Quota   | RAM Used   | Disk Used  | ARR |
        +---------+-----------+-----------------+----------+------------+-----+--------+-------------+------------+------------+-----+
        | bucket1 | couchbase | couchstore      | 0        | none       | 0   | 3000   | 1000.00 MiB | 55.32 MiB  | 92.04 MiB  | 100 |
        | bucket2 | couchbase | couchstore      | 0        | none       | 0   | 3000   | 1000.00 MiB | 55.15 MiB  | 101.29 MiB | 100 |
        | default | couchbase | couchstore      | 0        | none       | 0   | 380000 | 7.32 GiB    | 171.16 MiB | 143.26 MiB | 100 |
        +---------+-----------+-----------------+----------+------------+-----+--------+-------------+------------+------------+-----+

      • Update bucket replica=3 for all buckets and perform rebalance
      • Drop few collections and perform doc_loading in parallel to rebalance

      Observation:

      Rebalance completed successfully. But compaction tasks for the buckets "bucket1" and "default" got stuck.

       

       

      From .155 logs

      memcached.log.000000.txt:2022-02-13T22:36:09.761389-08:00 INFO (bucket1) Compaction of vb:127, task:6241, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.762895-08:00 INFO (default) Compaction of vb:53, task:6348, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.764182-08:00 INFO (default) Compaction of vb:204, task:6515, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.764309-08:00 INFO (default) Compaction of vb:31, task:6331, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.772640-08:00 INFO (default) Compaction of vb:57, task:6358, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.774104-08:00 INFO (default) Compaction of vb:35, task:6333, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.782016-08:00 INFO (default) Compaction of vb:39, task:6334, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.789916-08:00 INFO (default) Compaction of vb:43, task:6336, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.826149-08:00 INFO (bucket2) Compaction of vb:127, task:6137, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.887540-08:00 INFO (bucket1) Compaction of vb:156, task:6245, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:09.920422-08:00 INFO (bucket2) Compaction of vb:156, task:6142, purge_before_ts:0, purge_before_seq:0, drop_deletes:false, internal:true rescheduled with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:25.594486-08:00 INFO (bucket1) Compaction of vb:2, task:7086, purge_before_ts:1644561365, purge_before_seq:0, drop_deletes:false, internal:false created with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:25.594547-08:00 INFO (bucket1) Compaction of vb:1, task:7087, purge_before_ts:1644561365, purge_before_seq:0, drop_deletes:false, internal:false created with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:25.595407-08:00 INFO (bucket1) Compaction of vb:3, task:7088, purge_before_ts:1644561365, purge_before_seq:0, drop_deletes:false, internal:false created with delay:2147483647s (awaiting completion).
      memcached.log.000000.txt:2022-02-13T22:36:25.595963-08:00 INFO (bucket1) Compaction of vb:0, task:7089, purge_before_ts:1644561365, purge_before_seq:0, drop_deletes:false, internal:false created with delay:2147483647s (awaiting completion).

       

       

      Attachments

        1. test.log
          1.42 MB
          Ashwin Govindarajulu

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty