CHANGELOG for kv_engine (5.5.0-1758 to 5.5.0-1788):
* Commit: 3547c1d03cd4c0ae1ec2a8dd3c99222eb89e0020
MB-27793: Create script to convert Google Benchmark JSON response to XML
for CBNT
Example output from Google Benchmark:
{
"context": {
"date": "2018-01-30 15:05:01",
"num_cpus": 8,
"mhz_per_cpu": 2300,
"cpu_scaling_enabled": false,
"library_build_type": "release"
},
"benchmarks": [
{
"name": "AccessLogBenchEngine/MemoryOverhead/0/32768/min_time:0.000",
"iterations": 1,
"real_time": 43218,
"cpu_time": 9000,
"time_unit": "ns",
"MaxBytesAllocatedPerItem": 0,
"label": "Control"
},
{
"name": "AccessLogBenchEngine/MemoryOverhead/1/32768/min_time:0.000",
"iterations": 1,
"real_time": 33869176,
"cpu_time": 27173000,
"time_unit": "ns",
"MaxBytesAllocatedPerItem": 80,
"label": "AccessScanner"
},
{
"name": "DefragmentBench/Visit/0",
"iterations": 1,
"real_time": 857738169,
"cpu_time": 855854000,
"time_unit": "ns",
"ItemsPerSec": 5836006,
"label": "ValueOnly"
}
]
}
Example formatted output:
Change-Id: Ib75f221692e7c4c7df4a6c055228715984dca7ca
Reviewed-on: http://review.couchbase.org/88564
Reviewed-by: Dave Rigby
Tested-by: Tim Bradgate
* Commit: ed902fb49d0959fbb8db8e00d5a149c38aa29883
MB-27348 CouchRocks: Expose RocksDB Block Cache hit ratio stats
Beyond the raw Block Cache hit/miss value, we are more interested in the
Block Cache hit ratio for the data/index/filter blocks.
With this change we expose the new following metrics to 'cbstats':
'all' set:
- ep_rocksdb_block_cache_data_hit_ratio
- ep_rocksdb_block_cache_index_hit_ratio
- ep_rocksdb_block_cache_filter_hit_ratio
'kvstore' set:
- rocksdb_block_cache_data_hit_ratio
- rocksdb_block_cache_index_hit_ratio
- rocksdb_block_cache_filter_hit_ratio
We also remove the previous raw hit/miss stats from the 'all' set to
limit the size of stats on disk.
Change-Id: I7fa1e5b349ab5767af2c0305b6121cfe934bb0d6
Reviewed-on: http://review.couchbase.org/88421
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 3fe022f7a1f5f0df4e7135e80dbef4b4471543c1
Fix subdoc format description to match code
Change-Id: Ib930b068a0da40555dcad31deac81e30ec484f1c
Reviewed-on: http://review.couchbase.org/88545
Tested-by: Timofey Barmin
Reviewed-by: Trond Norbye
Tested-by: Build Bot
* Commit: 4f4de28c82dee03ca78b4708d0dfe9587965ba7e
MB-27779: Enable HELLO::Snappy feature negotiation
Allow clients to negotiate Hello::Snappy with the server. Upon
successful negotation, responses from the server whose value is SNAPPY
will set the datatype.SNAPPY bit
Change-Id: I588ce0050d06049c37a0666c97d51cd800652f92
Reviewed-on: http://review.couchbase.org/88504
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 2c5efbe7e1940805be9087d23ce18c4b855e7cbd
MB-27604: Don't rollback if start_seqno > purge_seqno > snap_start_seqno
We need a rollback due to purge inorder to not miss out on any
permanently deleted items. Currently our check for rollback due to
purge is very strict and we ask the client to rollback if we have purged
an item in the snapshot the client is looking for.
However to not miss out on any permanently deleted items, we should ask
the clients to rollback only if the client wants to start from a seqno
that is lesser than the purge_seqno. That is, only if
"start_seqno > purge_seqno > snap_start_seqno".
Change-Id: Ibfae86b35a4fd26efc5b96b350748b3bc4621f78
Reviewed-on: http://review.couchbase.org/87929
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 4f7b07d885b2a2066bcb5f297af6774aba697920
gcc7 warnings: algorithm may be uninitialized
The algorithm variable was set in the switch statement
handling all of the legal values for the scoped enum,
but the function could have been called with invalid
input.
Change-Id: I0833ed46ad02fa1dc57b88212bfb83a8b945d2c6
Reviewed-on: http://review.couchbase.org/88495
Reviewed-by: Tim Bradgate
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 8ac98438d5c05569c6380d375d28a38bb27eeb92
MB-27621: Stop server crash when trying to open an access log with an
invalid path
To do this we wrap the entire set up of the task in a try-catch and
log the exception produced if the creation fails. This allows the
task to be reshecduled at a later date.
Also adds an associated test.
Change-Id: I0ef2e833f431fee548bab477fe843ca9eb280477
Reviewed-on: http://review.couchbase.org/87980
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 9d5eebd4f818d885bdfdd48b8264e0106680510e
Refactor: Add a method to fetch the logger
A number of cmake targets had to be updated to reference
the memcached_logger library to have the correct include
path being propagated.
Change-Id: I36e905b57f86e714ec0b05c36497162b3b5804aa
Reviewed-on: http://review.couchbase.org/88365
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 82ac78ada51d82b32a00cf45aad44b4389f73668
Remove engine feature information
The feature information was never used and is currently
generating warnings from gcc 7. Given that we don't use
it we should nuke it.
Change-Id: Ica99af5e9dda549b090b0a4033b0f577e08a5438
Reviewed-on: http://review.couchbase.org/88494
Reviewed-by: Tim Bradgate
Tested-by: Build Bot
* Commit: 17ede5cf4318704400a02d26eb186024eab22b8c
Change the log levels in spdlogger
Previously we had NOTICE mapped to WARNING and WARNING to ERROR.
This makes the logs look a bit "odd", because what's logged
with NOTICE isn't a warning, and the warnings aren't erros.
Change the mapping so that NOTICE maps to INFO and WARNING
maps to WARNING.
Change-Id: I102027e85e60b5b1a79bc08a2ed202e6f5cfac87
Reviewed-on: http://review.couchbase.org/88486
Reviewed-by: Tim Bradgate
Tested-by: Build Bot
* Commit: 63c0d7f02d1f94ad39160e38bd7f55ff6b7196e7
MB-27617: Create a script to get 'cbmonitor' Perfrunner data
The 'get_cbmonitor_data.py' script downloads the raw cbmonitor data
for a given list of Perfrunner jobs and dumps some performance
metrics to file. The script craets a JSON file per each job and an
aggregated CSV file with data of all jobs.
Usage: python get_cbmonitor_data.py --job_list \
:[:''] [: ..] \
--output_dir
E.g.: python get_cbmonitor_data.py --job_list \
hera-pl:60 hera-pl:67:'RocksDB high ops' --output_dir .
The example above will create the following output files:
./hera-pl-60.json
./hera-pl-67.json
./data.csv
Change-Id: If2d44c83590e759ab5bdb9eec41fb039ee7cd92f
Reviewed-on: http://review.couchbase.org/88033
Tested-by: Build Bot
Reviewed-by: Dave Rigby
* Commit: 5578724b308fbb4849af2346d4dd86a4595064f8
MB-24860: Use CoreStore for totalMemory
Use the CoreStore container for the totalMemory stat
This commit has a performance affect which needs
'full-stack' validation, however here's the benchmark
output:
Run on (24 X 1217.34 MHz CPU s)
2018-01-24 09:58:21
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
MemoryAllocationStat/AllocNRead1/1/threads:96 46 ns 4090 ns 444576
MemoryAllocationStat/AllocNRead1/2/threads:96 72 ns 6496 ns 96000
MemoryAllocationStat/AllocNRead1/4/threads:96 75 ns 7020 ns 137856
MemoryAllocationStat/AllocNRead1/8/threads:96 68 ns 6564 ns 98304
MemoryAllocationStat/AllocNRead1/16/threads:96 74 ns 7117 ns 89568
MemoryAllocationStat/AllocNRead1/32/threads:96 72 ns 6737 ns 96000
MemoryAllocationStat/AllocNRead1/64/threads:96 61 ns 5858 ns 125568
MemoryAllocationStat/AllocNRead1/128/threads:96 184 ns 16677 ns 64512
MemoryAllocationStat/AllocNRead1/256/threads:96 137 ns 12700 ns 55008
MemoryAllocationStat/AllocNRead1/512/threads:96 239 ns 22485 ns 26208
MemoryAllocationStat/AllocNRead1/1024/threads:96 468 ns 44543 ns 16224
MemoryAllocationStat/AllocNRead1/2000/threads:96 919 ns 84507 ns 8832
Before numbers (logged from c02c7487)
Run on (24 X 1231.66 MHz CPU s)
2018-01-24 09:55:47
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
MemoryAllocationStat/AllocNRead1/1/threads:96 1 ns 111 ns 11745792
MemoryAllocationStat/AllocNRead1/2/threads:96 2 ns 178 ns 3904896
MemoryAllocationStat/AllocNRead1/4/threads:96 3 ns 286 ns 2148384
MemoryAllocationStat/AllocNRead1/8/threads:96 6 ns 522 ns 1237248
MemoryAllocationStat/AllocNRead1/16/threads:96 11 ns 974 ns 775296
MemoryAllocationStat/AllocNRead1/32/threads:96 21 ns 1921 ns 376224
MemoryAllocationStat/AllocNRead1/64/threads:96 30 ns 2853 ns 227328
MemoryAllocationStat/AllocNRead1/128/threads:96 75 ns 6358 ns 124608
MemoryAllocationStat/AllocNRead1/256/threads:96 128 ns 11724 ns 68640
MemoryAllocationStat/AllocNRead1/512/threads:96 236 ns 22184 ns 32640
MemoryAllocationStat/AllocNRead1/1024/threads:96 493 ns 45368 ns 14976
MemoryAllocationStat/AllocNRead1/2000/threads:96 1084 ns 88300 ns 8160
Change-Id: I9a60bc73fbf75e180fb8c294999143a6d3393d85
Reviewed-on: http://review.couchbase.org/87911
Reviewed-by: Daniel Owen
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: 4d213801243511c34befafd8485f7abe3f215fb4
MB-26729: Enforce MaxTTL on SetWithMeta
Add MaxTTL checking code on the setWithMeta path to ensure incoming
meta ops don't escape the MaxTTL config.
Change-Id: I223cd194b120e2ab5499013e4a669314bfb18837
Reviewed-on: http://review.couchbase.org/87979
Reviewed-by: Dave Rigby
Tested-by: Build Bot
* Commit: aac104fc11fbda858e3880a51c8d3c6101d1f77b
MB-22010: Link HdrHistogram_c library with ep_engine
Make the HdrHistogram_c header files available to ep_engine. Also make
the HdrHistogram_c library available to the ep_engine library and
ep-engine_ep_unit_tests.
Also includes a basic test which makes uses of the HdrHistogram_c
library functionality.
Note: The library is linked statically because when compiled for Windows
the library did not produce a .lib and hence could not be linked.
Change-Id: If67448aded7da489d674ef220f49ad672b334e27
Reviewed-on: http://review.couchbase.org/87973
Tested-by: Build Bot
Reviewed-by: Dave Rigby
* Commit: 44c7b290c55615a64604de2842fc6221719d96ea
MB-27063: Read audit version from the configuration file
Now that we are introducing version 2 of the audit configuration file,
we should read the version number from the configuration file and
set the appropriate value in the config. Then we need to read
the config value as opposed to just returning a hard-coded value.
Change-Id: Ibddf0096d3f0b744d1d63981cd64e921e343884e
Reviewed-on: http://review.couchbase.org/87906
Tested-by: Build Bot
Reviewed-by: Trond Norbye
* Commit: c02c74875bd5665a176e0b32506cbfd8ffb5d7cb
MB-24860: Add benchmark for memoryUsed stat
The memoryUsed stats is mutated and read with special code, it's not
a plain int!
* ThreadLocal storage
* Merging of counters to a 'global'
Add benchmarks to see how the stat tracking API performs
Sample output (running on mancouch):
Note: MemoryAllocationStat/AllocNRead1/4/threads:96, the 4 is the
number of calls to memAllocated, so 4 "allocs" to 1 getTotalMemoryUsed
Run on (24 X 1231.66 MHz CPU s)
2018-01-24 09:55:47
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
MemoryAllocationStat/AllocNRead1/1/threads:96 1 ns 111 ns 11745792
MemoryAllocationStat/AllocNRead1/2/threads:96 2 ns 178 ns 3904896
MemoryAllocationStat/AllocNRead1/4/threads:96 3 ns 286 ns 2148384
MemoryAllocationStat/AllocNRead1/8/threads:96 6 ns 522 ns 1237248
MemoryAllocationStat/AllocNRead1/16/threads:96 11 ns 974 ns 775296
MemoryAllocationStat/AllocNRead1/32/threads:96 21 ns 1921 ns 376224
MemoryAllocationStat/AllocNRead1/64/threads:96 30 ns 2853 ns 227328
MemoryAllocationStat/AllocNRead1/128/threads:96 75 ns 6358 ns 124608
MemoryAllocationStat/AllocNRead1/256/threads:96 128 ns 11724 ns 68640
MemoryAllocationStat/AllocNRead1/512/threads:96 236 ns 22184 ns 32640
MemoryAllocationStat/AllocNRead1/1024/threads:96 493 ns 45368 ns 14976
MemoryAllocationStat/AllocNRead1/2000/threads:96 1084 ns 88300 ns 8160
Change-Id: I708785d9ab603f6714d235b246d0b8dcd979140e
Reviewed-on: http://review.couchbase.org/87972
Tested-by: Build Bot
Reviewed-by: Daniel Owen
* Commit: 5707a56ba95e18fe844ba3c9717c1940468c68df
MB-27489: Add further debug logging when backfill fails to read disk file
This commit logs the vbucket state when the backfill fails to read the
vbucket file and also bumps up the log level of a vbucket being deleted.
Change-Id: I19c988787c1b15adc9d3a9564817743adca201a8
Reviewed-on: http://review.couchbase.org/88409
Tested-by: Build Bot
Reviewed-by: Dave Rigby
* Commit: e3bcabafd97f5703e7391ef7de9dfbf0f16b82fa
MB-23166: Remove redundant stats generated from config aliases
The changes in http://review.couchbase.org/#/c/74696/ changed the name
of the following stats:
ep_max_num_writers -> ep_num_writer_threads
ep_max_num_readers -> ep_num_reader_threads
ep_max_num_auxio -> ep_num_auxio_threads
ep_max_num_nonio -> ep_num_nonio_threads
but retained these names as aliases to allow for a transitionary
period of backward compatibility.
As these are redundant, removing from the next release after 5.0
Change-Id: I1ea2fc5d3a405926bacf2d545214ea3c0e1521ed
Reviewed-on: http://review.couchbase.org/88125
Reviewed-by: Tim Bradgate
Tested-by: Build Bot