CHANGELOG for kv_engine (5.5.0-1758 to 5.5.0-1788): * Commit: 3547c1d03cd4c0ae1ec2a8dd3c99222eb89e0020 MB-27793: Create script to convert Google Benchmark JSON response to XML for CBNT Example output from Google Benchmark: { "context": { "date": "2018-01-30 15:05:01", "num_cpus": 8, "mhz_per_cpu": 2300, "cpu_scaling_enabled": false, "library_build_type": "release" }, "benchmarks": [ { "name": "AccessLogBenchEngine/MemoryOverhead/0/32768/min_time:0.000", "iterations": 1, "real_time": 43218, "cpu_time": 9000, "time_unit": "ns", "MaxBytesAllocatedPerItem": 0, "label": "Control" }, { "name": "AccessLogBenchEngine/MemoryOverhead/1/32768/min_time:0.000", "iterations": 1, "real_time": 33869176, "cpu_time": 27173000, "time_unit": "ns", "MaxBytesAllocatedPerItem": 80, "label": "AccessScanner" }, { "name": "DefragmentBench/Visit/0", "iterations": 1, "real_time": 857738169, "cpu_time": 855854000, "time_unit": "ns", "ItemsPerSec": 5836006, "label": "ValueOnly" } ] } Example formatted output: Change-Id: Ib75f221692e7c4c7df4a6c055228715984dca7ca Reviewed-on: http://review.couchbase.org/88564 Reviewed-by: Dave Rigby Tested-by: Tim Bradgate * Commit: ed902fb49d0959fbb8db8e00d5a149c38aa29883 MB-27348 CouchRocks: Expose RocksDB Block Cache hit ratio stats Beyond the raw Block Cache hit/miss value, we are more interested in the Block Cache hit ratio for the data/index/filter blocks. With this change we expose the new following metrics to 'cbstats': 'all' set: - ep_rocksdb_block_cache_data_hit_ratio - ep_rocksdb_block_cache_index_hit_ratio - ep_rocksdb_block_cache_filter_hit_ratio 'kvstore' set: - rocksdb_block_cache_data_hit_ratio - rocksdb_block_cache_index_hit_ratio - rocksdb_block_cache_filter_hit_ratio We also remove the previous raw hit/miss stats from the 'all' set to limit the size of stats on disk. Change-Id: I7fa1e5b349ab5767af2c0305b6121cfe934bb0d6 Reviewed-on: http://review.couchbase.org/88421 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 3fe022f7a1f5f0df4e7135e80dbef4b4471543c1 Fix subdoc format description to match code Change-Id: Ib930b068a0da40555dcad31deac81e30ec484f1c Reviewed-on: http://review.couchbase.org/88545 Tested-by: Timofey Barmin Reviewed-by: Trond Norbye Tested-by: Build Bot * Commit: 4f4de28c82dee03ca78b4708d0dfe9587965ba7e MB-27779: Enable HELLO::Snappy feature negotiation Allow clients to negotiate Hello::Snappy with the server. Upon successful negotation, responses from the server whose value is SNAPPY will set the datatype.SNAPPY bit Change-Id: I588ce0050d06049c37a0666c97d51cd800652f92 Reviewed-on: http://review.couchbase.org/88504 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 2c5efbe7e1940805be9087d23ce18c4b855e7cbd MB-27604: Don't rollback if start_seqno > purge_seqno > snap_start_seqno We need a rollback due to purge inorder to not miss out on any permanently deleted items. Currently our check for rollback due to purge is very strict and we ask the client to rollback if we have purged an item in the snapshot the client is looking for. However to not miss out on any permanently deleted items, we should ask the clients to rollback only if the client wants to start from a seqno that is lesser than the purge_seqno. That is, only if "start_seqno > purge_seqno > snap_start_seqno". Change-Id: Ibfae86b35a4fd26efc5b96b350748b3bc4621f78 Reviewed-on: http://review.couchbase.org/87929 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 4f7b07d885b2a2066bcb5f297af6774aba697920 gcc7 warnings: algorithm may be uninitialized The algorithm variable was set in the switch statement handling all of the legal values for the scoped enum, but the function could have been called with invalid input. Change-Id: I0833ed46ad02fa1dc57b88212bfb83a8b945d2c6 Reviewed-on: http://review.couchbase.org/88495 Reviewed-by: Tim Bradgate Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 8ac98438d5c05569c6380d375d28a38bb27eeb92 MB-27621: Stop server crash when trying to open an access log with an invalid path To do this we wrap the entire set up of the task in a try-catch and log the exception produced if the creation fails. This allows the task to be reshecduled at a later date. Also adds an associated test. Change-Id: I0ef2e833f431fee548bab477fe843ca9eb280477 Reviewed-on: http://review.couchbase.org/87980 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 9d5eebd4f818d885bdfdd48b8264e0106680510e Refactor: Add a method to fetch the logger A number of cmake targets had to be updated to reference the memcached_logger library to have the correct include path being propagated. Change-Id: I36e905b57f86e714ec0b05c36497162b3b5804aa Reviewed-on: http://review.couchbase.org/88365 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 82ac78ada51d82b32a00cf45aad44b4389f73668 Remove engine feature information The feature information was never used and is currently generating warnings from gcc 7. Given that we don't use it we should nuke it. Change-Id: Ica99af5e9dda549b090b0a4033b0f577e08a5438 Reviewed-on: http://review.couchbase.org/88494 Reviewed-by: Tim Bradgate Tested-by: Build Bot * Commit: 17ede5cf4318704400a02d26eb186024eab22b8c Change the log levels in spdlogger Previously we had NOTICE mapped to WARNING and WARNING to ERROR. This makes the logs look a bit "odd", because what's logged with NOTICE isn't a warning, and the warnings aren't erros. Change the mapping so that NOTICE maps to INFO and WARNING maps to WARNING. Change-Id: I102027e85e60b5b1a79bc08a2ed202e6f5cfac87 Reviewed-on: http://review.couchbase.org/88486 Reviewed-by: Tim Bradgate Tested-by: Build Bot * Commit: 63c0d7f02d1f94ad39160e38bd7f55ff6b7196e7 MB-27617: Create a script to get 'cbmonitor' Perfrunner data The 'get_cbmonitor_data.py' script downloads the raw cbmonitor data for a given list of Perfrunner jobs and dumps some performance metrics to file. The script craets a JSON file per each job and an aggregated CSV file with data of all jobs. Usage: python get_cbmonitor_data.py --job_list \ :[:''] [: ..] \ --output_dir E.g.: python get_cbmonitor_data.py --job_list \ hera-pl:60 hera-pl:67:'RocksDB high ops' --output_dir . The example above will create the following output files: ./hera-pl-60.json ./hera-pl-67.json ./data.csv Change-Id: If2d44c83590e759ab5bdb9eec41fb039ee7cd92f Reviewed-on: http://review.couchbase.org/88033 Tested-by: Build Bot Reviewed-by: Dave Rigby * Commit: 5578724b308fbb4849af2346d4dd86a4595064f8 MB-24860: Use CoreStore for totalMemory Use the CoreStore container for the totalMemory stat This commit has a performance affect which needs 'full-stack' validation, however here's the benchmark output: Run on (24 X 1217.34 MHz CPU s) 2018-01-24 09:58:21 ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. ---------------------------------------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------------------------------------- MemoryAllocationStat/AllocNRead1/1/threads:96 46 ns 4090 ns 444576 MemoryAllocationStat/AllocNRead1/2/threads:96 72 ns 6496 ns 96000 MemoryAllocationStat/AllocNRead1/4/threads:96 75 ns 7020 ns 137856 MemoryAllocationStat/AllocNRead1/8/threads:96 68 ns 6564 ns 98304 MemoryAllocationStat/AllocNRead1/16/threads:96 74 ns 7117 ns 89568 MemoryAllocationStat/AllocNRead1/32/threads:96 72 ns 6737 ns 96000 MemoryAllocationStat/AllocNRead1/64/threads:96 61 ns 5858 ns 125568 MemoryAllocationStat/AllocNRead1/128/threads:96 184 ns 16677 ns 64512 MemoryAllocationStat/AllocNRead1/256/threads:96 137 ns 12700 ns 55008 MemoryAllocationStat/AllocNRead1/512/threads:96 239 ns 22485 ns 26208 MemoryAllocationStat/AllocNRead1/1024/threads:96 468 ns 44543 ns 16224 MemoryAllocationStat/AllocNRead1/2000/threads:96 919 ns 84507 ns 8832 Before numbers (logged from c02c7487) Run on (24 X 1231.66 MHz CPU s) 2018-01-24 09:55:47 ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. ---------------------------------------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------------------------------------- MemoryAllocationStat/AllocNRead1/1/threads:96 1 ns 111 ns 11745792 MemoryAllocationStat/AllocNRead1/2/threads:96 2 ns 178 ns 3904896 MemoryAllocationStat/AllocNRead1/4/threads:96 3 ns 286 ns 2148384 MemoryAllocationStat/AllocNRead1/8/threads:96 6 ns 522 ns 1237248 MemoryAllocationStat/AllocNRead1/16/threads:96 11 ns 974 ns 775296 MemoryAllocationStat/AllocNRead1/32/threads:96 21 ns 1921 ns 376224 MemoryAllocationStat/AllocNRead1/64/threads:96 30 ns 2853 ns 227328 MemoryAllocationStat/AllocNRead1/128/threads:96 75 ns 6358 ns 124608 MemoryAllocationStat/AllocNRead1/256/threads:96 128 ns 11724 ns 68640 MemoryAllocationStat/AllocNRead1/512/threads:96 236 ns 22184 ns 32640 MemoryAllocationStat/AllocNRead1/1024/threads:96 493 ns 45368 ns 14976 MemoryAllocationStat/AllocNRead1/2000/threads:96 1084 ns 88300 ns 8160 Change-Id: I9a60bc73fbf75e180fb8c294999143a6d3393d85 Reviewed-on: http://review.couchbase.org/87911 Reviewed-by: Daniel Owen Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: 4d213801243511c34befafd8485f7abe3f215fb4 MB-26729: Enforce MaxTTL on SetWithMeta Add MaxTTL checking code on the setWithMeta path to ensure incoming meta ops don't escape the MaxTTL config. Change-Id: I223cd194b120e2ab5499013e4a669314bfb18837 Reviewed-on: http://review.couchbase.org/87979 Reviewed-by: Dave Rigby Tested-by: Build Bot * Commit: aac104fc11fbda858e3880a51c8d3c6101d1f77b MB-22010: Link HdrHistogram_c library with ep_engine Make the HdrHistogram_c header files available to ep_engine. Also make the HdrHistogram_c library available to the ep_engine library and ep-engine_ep_unit_tests. Also includes a basic test which makes uses of the HdrHistogram_c library functionality. Note: The library is linked statically because when compiled for Windows the library did not produce a .lib and hence could not be linked. Change-Id: If67448aded7da489d674ef220f49ad672b334e27 Reviewed-on: http://review.couchbase.org/87973 Tested-by: Build Bot Reviewed-by: Dave Rigby * Commit: 44c7b290c55615a64604de2842fc6221719d96ea MB-27063: Read audit version from the configuration file Now that we are introducing version 2 of the audit configuration file, we should read the version number from the configuration file and set the appropriate value in the config. Then we need to read the config value as opposed to just returning a hard-coded value. Change-Id: Ibddf0096d3f0b744d1d63981cd64e921e343884e Reviewed-on: http://review.couchbase.org/87906 Tested-by: Build Bot Reviewed-by: Trond Norbye * Commit: c02c74875bd5665a176e0b32506cbfd8ffb5d7cb MB-24860: Add benchmark for memoryUsed stat The memoryUsed stats is mutated and read with special code, it's not a plain int! * ThreadLocal storage * Merging of counters to a 'global' Add benchmarks to see how the stat tracking API performs Sample output (running on mancouch): Note: MemoryAllocationStat/AllocNRead1/4/threads:96, the 4 is the number of calls to memAllocated, so 4 "allocs" to 1 getTotalMemoryUsed Run on (24 X 1231.66 MHz CPU s) 2018-01-24 09:55:47 ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. ---------------------------------------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------------------------------------- MemoryAllocationStat/AllocNRead1/1/threads:96 1 ns 111 ns 11745792 MemoryAllocationStat/AllocNRead1/2/threads:96 2 ns 178 ns 3904896 MemoryAllocationStat/AllocNRead1/4/threads:96 3 ns 286 ns 2148384 MemoryAllocationStat/AllocNRead1/8/threads:96 6 ns 522 ns 1237248 MemoryAllocationStat/AllocNRead1/16/threads:96 11 ns 974 ns 775296 MemoryAllocationStat/AllocNRead1/32/threads:96 21 ns 1921 ns 376224 MemoryAllocationStat/AllocNRead1/64/threads:96 30 ns 2853 ns 227328 MemoryAllocationStat/AllocNRead1/128/threads:96 75 ns 6358 ns 124608 MemoryAllocationStat/AllocNRead1/256/threads:96 128 ns 11724 ns 68640 MemoryAllocationStat/AllocNRead1/512/threads:96 236 ns 22184 ns 32640 MemoryAllocationStat/AllocNRead1/1024/threads:96 493 ns 45368 ns 14976 MemoryAllocationStat/AllocNRead1/2000/threads:96 1084 ns 88300 ns 8160 Change-Id: I708785d9ab603f6714d235b246d0b8dcd979140e Reviewed-on: http://review.couchbase.org/87972 Tested-by: Build Bot Reviewed-by: Daniel Owen * Commit: 5707a56ba95e18fe844ba3c9717c1940468c68df MB-27489: Add further debug logging when backfill fails to read disk file This commit logs the vbucket state when the backfill fails to read the vbucket file and also bumps up the log level of a vbucket being deleted. Change-Id: I19c988787c1b15adc9d3a9564817743adca201a8 Reviewed-on: http://review.couchbase.org/88409 Tested-by: Build Bot Reviewed-by: Dave Rigby * Commit: e3bcabafd97f5703e7391ef7de9dfbf0f16b82fa MB-23166: Remove redundant stats generated from config aliases The changes in http://review.couchbase.org/#/c/74696/ changed the name of the following stats: ep_max_num_writers -> ep_num_writer_threads ep_max_num_readers -> ep_num_reader_threads ep_max_num_auxio -> ep_num_auxio_threads ep_max_num_nonio -> ep_num_nonio_threads but retained these names as aliases to allow for a transitionary period of backward compatibility. As these are redundant, removing from the next release after 5.0 Change-Id: I1ea2fc5d3a405926bacf2d545214ea3c0e1521ed Reviewed-on: http://review.couchbase.org/88125 Reviewed-by: Tim Bradgate Tested-by: Build Bot