Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.2.0
-
Untriaged
-
0
-
Unknown
Description
I ran some local testing and noted that the retained history was quite skewed, some vbuckets had history and some had none.
The following describes what I did and what I observed. This MB relates to the issue that after I finish the workload not all vbuckets are retaining history and the reported disk used seems quite high.
A second question comes up regarding manual compaction - does it have to discard all history (again disk used looks high after the compaction).
1) Simple setup using cluster_run
- COUCHBASE_NUM_VBUCKETS=16
- Start a single node
- Create a single magma bucket, this happens to have 3000 MB quota (just the last cluster_connect in my shell history).
2) Checked that history is on for the default collection.
> ./cbstats localhost:12000 -u Administrator -p asdasd -b default collections | grep hist
|
0x0:0x0:history: true <-- good!
|
3) Configured retention by size only, I wanted a smallish size per vbucket, so set 4,194,304 (262,144 per vb)
curl localhost:9000/pools/default/buckets/default -u Administrator:asdasd -X POST -d historyRetentionBytes=4194304
|
4) Let the system settle and checked the baseline disk used for the bucket (looking at the UI)
- UI shows 170KiB
5) Perform some insert/update of a few keys. I was sort of aiming for 1 key per vbucket (not easy) so aimed for 2 per VB, however pillowfight gave a distribution of 1/2/3 per VB. The workload did hit every vbucket though and note the document size is 1024 and random body is on to hinder compression.
> timeout 5s ./cbc-pillowfight -u Administrator -P asdasd -U 'couchbase://localhost:12000/default' -I 32 -m 1024 -M 1024 -R
|
6) I checked that each vbucket has seen a good amount of churn (so assume lots of updates are retained) and each should of hit the retention limit of 262,144.
./cbstats localhost:12000 -u Administrator -p asdasd -b default vbucket-details | grep high_seq
|
vb_0:high_seqno: 15382
|
vb_1:high_seqno: 7651
|
vb_2:high_seqno: 15351
|
vb_3:high_seqno: 15348
|
vb_4:high_seqno: 15337
|
vb_5:high_seqno: 15320
|
vb_6:high_seqno: 7686
|
vb_7:high_seqno: 15411
|
vb_8:high_seqno: 23027
|
vb_9:high_seqno: 23021
|
vb_10:high_seqno: 15383
|
vb_11:high_seqno: 7726
|
vb_12:high_seqno: 7697
|
vb_13:high_seqno: 15367
|
vb_14:high_seqno: 23028
|
vb_15:high_seqno: 23026
|
7) Next check each vbucket has retained history, this is where things get surprising.
./cbstats localhost:12000 -u Administrator -p asdasd -b default vbucket-details | grep history
|
vb_0:history_start_seqno: 2
|
vb_1:history_start_seqno: 2
|
vb_2:history_start_seqno: 9600
|
vb_3:history_start_seqno: 9643
|
vb_4:history_start_seqno: 0 <- ?????
|
vb_5:history_start_seqno: 0
|
vb_6:history_start_seqno: 0
|
vb_7:history_start_seqno: 0
|
vb_8:history_start_seqno: 17802
|
vb_9:history_start_seqno: 17728
|
vb_10:history_start_seqno: 0
|
vb_11:history_start_seqno: 0
|
vb_12:history_start_seqno: 2
|
vb_13:history_start_seqno: 2
|
vb_14:history_start_seqno: 0
|
vb_15:history_start_seqno: 0
|
I understand why some vbuckets have a huge history_start_seqno, but why do some have nothing? KV uses a value of zero to mean no history exists.
For example vb:14 has 3 items but no history.
> ./cbstats localhost:12000 -u Administrator -p asdasd -b default vbucket-details 14 | grep -e num_items -e history
|
vb_14:history_start_seqno: 0
|
vb_14:ht_num_items: 3
|
vb_14:num_items: 3
|
8) Next re-check the disk usage from the UI
- 61.3MiB
Ok.. so manual compact and look again.
- 34.6MiB
That seems high as we now should only have 32 copies of the 1024 documents? Maybe there's more overhead in magma than i expect?
Manual compaction also dropped ALL history which may surprise the user (it surprised me)
> ./cbstats localhost:12000 -u Administrator -p asdasd -b default vbucket-details | grep history
|
vb_0:history_start_seqno: 0
|
vb_1:history_start_seqno: 0
|
vb_2:history_start_seqno: 0
|
vb_3:history_start_seqno: 0
|
vb_4:history_start_seqno: 0
|
vb_5:history_start_seqno: 0
|
vb_6:history_start_seqno: 0
|
vb_7:history_start_seqno: 0
|
vb_8:history_start_seqno: 0
|
vb_9:history_start_seqno: 0
|
vb_10:history_start_seqno: 0
|
vb_11:history_start_seqno: 0
|
vb_12:history_start_seqno: 0
|
vb_13:history_start_seqno: 0
|
vb_14:history_start_seqno: 0
|
vb_15:history_start_seqno: 0
|
Attachments
Issue Links
- is duplicated by
-
MB-55966 CDC: Enforce a minimum history_retention_bytes value of 2GiB
- Closed