CHANGELOG for asterixdb * Commit: cbfeef887cccbe9197c7d4517c9f54444e9e4835 in build: 6.5.0-4279 Merge commit '82b067c' from stabilization-f69489 Change-Id: I4891035d517387148e86027a733afa4a21a5c75a * Commit: 82b067cf0ac6c122802006e2220504e0176cdbc2 in build: 6.5.0-4279 [ASTERIXDB-2633][FUN] Fix LIKE to regex translation - user model changes: no - storage format changes: no - interface changes: no Change-Id: I6138f62fdcd60bcc641f662d1fdb05b46dafbebf Reviewed-on: https://asterix-gerrit.ics.uci.edu/3558 Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Murtadha Hubail <mhubail@apache.org> * Commit: 34915843d619e07b95f34458ae2e0992f44ff379 in build: 6.5.0-4273 Merge commit '8930f0d' from stabilization-f69489 Change-Id: I64cf862bc5ebe19a24acbccf821f68556699222e * Commit: 8930f0d7614f43ad52624fd121de45ab27fb281e in build: 6.5.0-4273 [NO ISSUE][ING][EXT] Handle filters on meta() fields for feeds - user model changes: no - storage format changes: no - interface changes: no Details: This fix is to handle filters on meta() fields where the feed name is not supplied as an arguement to the meta() when connecting the feed. Both of these statement should work: CONNECT FEED ds_feed TO DATASET ds WHERE meta(ds_feed).id LIKE "%WEB%" CONNECT FEED ds_feed TO DATASET ds WHERE meta().id LIKE "%WEB%" Change-Id: I1dbf40525601584efa29eb3adf4ba41062ebe53a Reviewed-on: https://asterix-gerrit.ics.uci.edu/3557 Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> * Commit: a9e99928357aac31af0a02b902ee0f9f9ebfc9f8 in build: 6.5.0-4267 [NO ISSUE][COMP] Remove listify() when iterating over aggregate - user model changes: no - storage format changes: no - interface changes: no Details: - Optimizer must remove listify() when iterating over an aggregate function that returns a list Change-Id: I7a8919595be7e4a44a5816ac7f281842d03ecc1f Reviewed-on: https://asterix-gerrit.ics.uci.edu/3556 Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com> * Commit: aa11886d2f8d40a25d51639291e1ec7b54004028 in build: 6.5.0-4264 [ASTERIXDB-2631][COMP][RT] Limit number of reported warnings - user model changes: no - storage format changes: no - interface changes: yes Details: Limit number of reported warnings. - added a new compiler property "COMPILER.RUNTIME.WARNINGS" with default value set to 0. - updated Stat class to have a field for the count of runtime warnings. - updated few classes to propagate the warnings limit number to Hyracks Task class. - added new Option to parse unsigned long values. - TaskProfile has a field for the count. - updated TestExecutor to allow specifying parameters in the body of the POST method. Change-Id: Ie8756f1b9dcb7bbce92bd77692490fbada9a2482 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3535 Reviewed-by: Murtadha Hubail <mhubail@apache.org> Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> * Commit: c2545cc1d8d100ac1765641eecf154bb68285e72 in build: 6.5.0-4255 [NO ISSUE][RT] Add Thread-Based Stats Collector - user model changes: no - storage format changes: no - interface changes: yes Details: - Add infra to allow collecting thread-based stats during runtime for any thread that belongs to a task. - Collect number of pinned pages per thread and report it in the TaskProfile. - Aggregate pinned pages counters from all job tasks and report it as diskIoCount in the metrics field in the json response. The plan is to move this stats to the profile field when it is introduced. - Collecting pinned pages stats is currently enabled by default for any job with IndexSearchOperatorNodePushable. The plan is to allow enabling/disabling as part of the job profiling change. - Add test case for diskIoCount metric. - Remove unused IndexSearchOperatorNodePushable constructor. Change-Id: I44dfcedcadb3d0f48815b521e7d495e473b02e3d Reviewed-on: https://asterix-gerrit.ics.uci.edu/3555 Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Murtadha Hubail <mhubail@apache.org> Reviewed-by: Till Westmann <tillw@apache.org> * Commit: 309b96f8d7a266c0d9e986bb294f611c065c4909 in build: 6.5.0-4241 [NO ISSUE][COMP] Increase memory budget for window operator - user model changes: no - storage format changes: no - interface changes: no Details: - Increase default memory budget for a window operator from 4MB to 32MB Change-Id: I6902d5e730232892f22e36598389309e8bc909c3 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3551 Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com> * Commit: 1e90caca704c85ba49f88b810163ae6fab344cf1 in build: 6.5.0-4240 [NO ISSUE][COMP] Reduce memory requirements for monitoring functions - user model changes: no - storage format changes: no - interface changes: no Details: - Queries that only invoke monitoring/metadata datasource functions should run with minimal memory requirements - The functions are: active_requests(), completed_requests(), dataset_resources(), storage_components(), jobs(), and ping() Change-Id: If7de53a5b476c8ce0d3fe485bf526a5d425e4c37 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3548 Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> Reviewed-by: Ali Alsuliman <ali.al.solaiman@gmail.com> * Commit: d292d99bc652e81ae1e7877e88529ce2b29cf6d0 in build: 6.5.0-4273 [NO ISSUE][COMP] Fix index selection for datasets with meta - user model changes: no - storage format changes: no - interface changes: no Details: Fix index selection for datasets with meta. Access method rule should check where the field is coming from (dataset record or meta record) and then determine if the field matches the keys in the index based on their names and sources. This patch also fixes resolving PK field accesses to the primary key variable (e.g. $ds.getField("id") is turned into $13 where id is a PK). The fix considers whether the PK is coming from the data record or the meta record. The patch also includes fixing rewriting of meta() references and replacing them with their corresponding meta variables. Now nested plans are visited also when looking for meta() references to take care of cases where the data scan producing the meta variable and the meta() references happen to be inside the nested plans. MetaFunctionToMetaVariable() is fired also after the rules which eliminate subplans to allow for rewriting of the meta() if it couldn't be replaced when the meta() reference was in the subplan but referring to meta variable outside the nested plan. ReinferAllTypesRule() is now fired before ByNameToByIndexFieldAccessRule() to allow the latter rule to get the up-to-date types in the whole plan. Change-Id: I0503f64cd51153896e2d7d7abc465c679f82e2fd Reviewed-on: https://asterix-gerrit.ics.uci.edu/3545 Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> Contrib: Till Westmann <tillw@apache.org> * Commit: 8154097da71086a4e38f475106e4f67115b1a07b in build: 6.5.0-4222 [NO ISSUE][STO] Increase Dataset Checkpoint Interval - user model changes: no - storage format changes: no - interface changes: no Details: - The current dataset checkpoint interval of 10 mins is too small and can result in many unnecessary small flushes. This change increases the interval to 60 mins. Change-Id: Idd4d1d02e4df83833220bca05193c6a99393ac99 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3546 Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Murtadha Hubail <mhubail@apache.org> Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Hussain Towaileb <hussainht@gmail.com> * Commit: cba9cbe380e04a183caadc13497c97d0db6ff31c in build: 6.5.0-4273 [NO ISSUE][CLUS] Attempt To Get Node IP From Existing Nodes - user model changes: no - storage format changes: no - interface changes: no Details: - When we fail to resolve a node's host to an IP on node failure/removal, attempt to find its IP from the list of existing nodes. This is done to prevent failing the node removal operation when the node's host address cannot be resolved anymore. Change-Id: I8f12a3dbb84a4cb731f1379efd65ca50606a8b07 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3544 Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Till Westmann <tillw@apache.org> * Commit: fd0f4ce8fe94d2837552d8cf09cdaf27b5a11f2f in build: 6.5.0-4273 [NO ISSUE][*DB][REPL] Identifier comparability of unknown hosts Ensure that replication identifiers comparisons are not influenced by address resolution Change-Id: I6d7aa87777d17a3416d16cfad23b2773b1614a02 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3536 Sonar-Qube: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Michael Blow <mblow@apache.org> Reviewed-by: Murtadha Hubail <mhubail@apache.org> * Commit: 9554871aaee87a3bd5e528123eca50aa212749af in build: 6.5.0-4273 [NO ISSUE][NET] Ensure Recycling Buffer and Notifying Sender is Atomic - user model changes: no - storage format changes: no - interface changes: no Details: - To avoid synchronization issues that might occur due to JVM reordering, ensure that both recycling read buffers and notifying the sender of their availability are done atomically before the next buffer is received from the sender. Change-Id: Ia3b1920f33bf7d4e7efbd2ea3405cbc4310a78c7 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3520 Contrib: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Murtadha Hubail <mhubail@apache.org> Reviewed-by: Till Westmann <tillw@apache.org> (cherry picked from commit 32eed5f384c5851eae1c613fcb3b9532744ed595) Reviewed-on: https://asterix-gerrit.ics.uci.edu/3525 Reviewed-by: Michael Blow <mblow@apache.org> CHANGELOG for backup * Commit: 6ccc60a1027b4a6bf9fdf5f78cdc733b67f55c98 in build: 6.5.0-4282 MB-35905 Avoid fetching the VBucket item count multiple times Change-Id: Ib3ab737ab630dd35607c4978e8438cb27a705e39 Reviewed-on: http://review.couchbase.org/114711 Reviewed-by: Carlos Gonzalez <carlos.gonzalez@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 7b1874dae372762c490240ed6c3620c94ae88666 in build: 6.5.0-4281 MB-35106 More informative disk full errors Change-Id: Id4aaeef6b0c62efd35baa5b6fe3f2f70e0ee76b9 Reviewed-on: http://review.couchbase.org/114700 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Patrick Varley <patrick@couchbase.com> * Commit: 4df2de7b3419ab63b22385812c784761183db1c4 in build: 6.5.0-4281 MB-35947 Print warning when TXN data is removed Change-Id: I272788ff7a1359e223326e1b999470c18492fe5d Reviewed-on: http://review.couchbase.org/114674 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Patrick Varley <patrick@couchbase.com> * Commit: 36b1dd2d721409383efaf92b0426b311b364317d in build: 6.5.0-4260 MB-35895 Use same date format for gocbcore logging Change-Id: I6943573ac960e4e5ae52d5bd4f1048810184c1ab Reviewed-on: http://review.couchbase.org/114592 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Patrick Varley <patrick@couchbase.com> * Commit: bfe0007150e7acfd99f7ee367e39a7b0d474d72a in build: 6.5.0-4260 MB-35892 Upgrade to Golang 1.13 Change-Id: If742d12607a161771c503d49f3cbea2982d00f9a Reviewed-on: http://review.couchbase.org/114597 Reviewed-by: Carlos Gonzalez <carlos.gonzalez@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 3dc6d9d1e7afb1566a82447a89cfc2bf6e283226 in build: 6.5.0-4257 MB-35863 Make restore stats consistent and non-locking Before the stats where all added using locks to guarantee consistency this meant that when using high number of threads this caused contention. When the restore worker pool was re-designed the locks were removed this meant that the stats where not consistent and thus sometimes where wrong. To fix this, each worker now keeps tracks of its own progress and an aggregator goroutine will aggregate the progress of each worker and create the final correct stats without locking. Change-Id: Ia23b32e89fb964f6abba7817765459219602b1c8 Reviewed-on: http://review.couchbase.org/114512 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: James Lee <james.lee@couchbase.com> * Commit: 80c6eab17d895ebef18ec8a7b44507c449fbebe9 in build: 6.5.0-4246 MB-35893 Do not assume that the server has 1024 vBuckets Couchbase Server on MacOS only supports using 64 vBuckets, therefore, we need to get the number of vBuckets from the server before calculating the amount of items remaining. Change-Id: I5c64314e60e6166337f5b9f86ac1a89daa5c61ec Reviewed-on: http://review.couchbase.org/114529 Reviewed-by: Carlos Gonzalez <carlos.gonzalez@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 378e2c69f73d9bd2dceefc60ce279a266d500098 in build: 6.5.0-4231 MB-35828 Use the BRStat struct when getting restore stats Change-Id: I74ce0b5f95797f3f88e72b277d663ff5173a32b5 Reviewed-on: http://review.couchbase.org/114349 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Carlos Gonzalez <carlos.gonzalez@couchbase.com> CHANGELOG for build * Commit: 95a61b3eae71654578e4ae40f3bb17791e758d8d in build: 6.5.0-4253 MB-34977: Add option to GPG sign Server RPM builds This presumes that the named GPG key is already loaded onto the build slave using "gpg --import" or similar. Change-Id: If63327213f310058c2b8e091c7236c01409ce893 Reviewed-on: http://review.couchbase.org/114513 Reviewed-by: Chris Hillery <ceej@couchbase.com> Tested-by: Chris Hillery <ceej@couchbase.com> CHANGELOG for cbas-core * Commit: 7c1945776bd4734ec569916232a47a96eb40e768 in build: 6.5.0-4273 Merge commit '8307d57' from alice Change-Id: I0d3d94334567896d60db68cbee39c815902f6de7 * Commit: 8307d572e8cc3aa09fd0e13decea4a3776e44860 in build: 6.5.0-4273 MB-35940: Analytics: Unable to filter Dataset on document keys - user model changes: no - storage format changes: no - interface changes: no Details: Backporting the fix. Add test cases for the filtering on the meta() fix. The fix is to handle filters on meta() fields where the bucket name is not supplied as an arguement to the meta() when creating a dataset and later connecting the bucket. Change-Id: I345e8d9bb8888f31454e471c3ab0adff3f5a8f57 Reviewed-on: http://review.couchbase.org/114637 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> * Commit: 69f244518d52047f3df43dd8014f7fe51c8c69fb in build: 6.5.0-4264 MB-34953: Limit number of reported warnings Change-Id: I57014e2e8eeaa53c09267a9af20d84fc9cd8e86a Reviewed-on: http://review.couchbase.org/113996 Reviewed-by: Murtadha Hubail <Murtadha.hubail@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 1b3ad313c25173f73b28a48d11fb8013bb331bf5 in build: 6.5.0-4258 Revert "MB-3443: Extend test framework to access N1QL endpoint" This reverts commit ff2e62749fe19eac1c8b3aaffcab0c790c6e0e68. Change-Id: I8ff3e5f9aba12a38c4ecad95a68a51b0bdc6a89d Reviewed-on: http://review.couchbase.org/114586 Reviewed-by: <sandeep.gupta@couchbase.com> Tested-by: <sandeep.gupta@couchbase.com> * Commit: ff2e62749fe19eac1c8b3aaffcab0c790c6e0e68 in build: 6.5.0-4257 MB-3443: Extend test framework to access N1QL endpoint - Queries within test files with ".cbquery.n1ql" are posted to n1ql endpoint; its output compared with corresponding expected result Change-Id: I1d777e17b4796941a57b5e8e1fa16e0adc49a8f0 Reviewed-on: http://review.couchbase.org/112842 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Till Westmann <till@couchbase.com> * Commit: 6c807e646b671f9a1f39211213cb01297d1a96ae in build: 6.5.0-4255 MB-35866: Ignore Metrics Fields Comparison in Backup Tests - Do not try to match the result metrics field by field in the backup/restore tests since they have no impact on the actual behavior of the test. This will eliminate the need to update the tests expected results when the metrics fields change. Change-Id: Ifea034178e983668514c6c590f03308aba376b9b Reviewed-on: http://review.couchbase.org/114543 Reviewed-by: Ali Alsuliman <ali.alsuliman@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: b2d465b49386741684e9be6147e9eac9822f24d7 in build: 6.5.0-4238 TXNJ-107: Make txn metadata more consistently named Change-Id: Iaf9ca9f27313624c52794028352a9bd1dfc181ba Reviewed-on: http://review.couchbase.org/113995 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Murtadha Hubail <Murtadha.hubail@couchbase.com> * Commit: 5624533e40aac510c8714dcba426b4f6c5b96c44 in build: 6.5.0-4273 MB-35809: Incorrect index chosen when field name is 'id' Fix index selection when fields happen to have the same names as the ones in meta(), e.g. "id" - added test cases Change-Id: I9f2880439933f69cf12358ded4e4025f865b4dd7 Reviewed-on: http://review.couchbase.org/114373 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Dmitry Lychagin <dmitry.lychagin@couchbase.com> CHANGELOG for cbas-ui * Commit: 47176289f752a4f5945e59e4bb72a30f8b9eec14 in build: 6.5.0-4276 Remove reference to obsolete code that is being removed. Change-Id: I8783d8ce30f9d71b8bfc72ed1cb1df2bb97a98b8 Reviewed-on: http://review.couchbase.org/114721 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 3993b99c8c9141e8e009fb598228d3158caefb02 in build: 6.5.0-4269 MB-35925: Refresh Returned Warnings on Each Request - Ensure the list of warnings displayed is refreshed with each request. - Allow warnings tooltip to grow upto 1024px. - Add a label at the begining of the warnings tooltip indicating how many warnings are displayed out of the total that was encountered during execution. Change-Id: Ibdc7fa85f4623d7a0d3e93c39cefeeb684c7fcf7 Reviewed-on: http://review.couchbase.org/114642 Reviewed-by: Murtadha Hubail <Murtadha.hubail@couchbase.com> Reviewed-by: Till Westmann <till@couchbase.com> Tested-by: Till Westmann <till@couchbase.com> CHANGELOG for couchbase-cli * Commit: cdfa3fdc9948c6cc996fb2570e1dd5ba13354ef7 in build: 6.5.0-4281 MB-35948 Print warning about TXN data being skipped Change-Id: I2b30f08c2797f55d9b99447859f9c1c3435ac976 Reviewed-on: http://review.couchbase.org/114682 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Patrick Varley <patrick@couchbase.com> * Commit: e60949d34fd78908bc56ac38f43c7853a68bdab2 in build: 6.5.0-4271 MB-35943 Fix index out of range error The skipped list is used to avoid waiting for a response for a message that was not sent, but before checking the next element of the list we have to confirm that there is a next element. Change-Id: Ib442249baa317a4baeb4520213bc83073bcf69a6 Reviewed-on: http://review.couchbase.org/114668 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: James Lee <james.lee@couchbase.com> * Commit: c457d553b4b03c4b1faf137d9a5104f4ed4870f2 in build: 6.5.0-4259 MB-35894 TXN backup/restore support Changes where made to the design of the xattrs as well as the key names for the transaction related data and so the restore transaction filtering had to be updated accordingly. Change-Id: Ic635e39052b8ff6b34645fa8c5c71dc4525e8078 Reviewed-on: http://review.couchbase.org/114587 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Patrick Varley <patrick@couchbase.com> CHANGELOG for couchdb * Commit: 9d18c70682a9d4d310d8745aed15699eaad14a2d in build: 6.5.0-4259 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci-views.northscale.in/views-11.09.2019-05.30.pass.html Change-Id: Ia34f6c0e2dc38adb546ea0043d54fb6710fcc6b2 * Commit: aa50310141c2a304079c082208247966573ee736 in build: 6.5.0-4259 MB-35814: Handle not_found case in update listener Change-Id: I687012342ab29a18773474befcd085814afc6230 Reviewed-on: http://review.couchbase.org/114431 Tested-by: <ankit.prabhu@couchbase.com> Reviewed-by: Suraj Naik <suraj.naik@couchbase.com> * Commit: 262157dc5071b2cb6ba3eff655fdcc52c55e1faa in build: 6.5.0-4230 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci-views.northscale.in/views-09.09.2019-02.42.pass.html Change-Id: I54e7e4f7b14576404aa0acbe0648b4c1ca0a2041 * Commit: 76a752304146ae07e7f7f12c9132f96951297829 in build: 6.5.0-4230 MB-35250: Log improvements during retry of request_group sample log message when request_replica_group fails for 1st time: "Retrying group `_design/try` (prod) request, stale=update_after, set `sample`" sample log message after it succeeds doing retries: "Request replica group succeeded for `_design/try` (prod) request, stale=update_after, set `sample` after 10 retry attempt" Change-Id: I438a6ef21662115e92546f4be0b87e9019d41f44 Reviewed-on: http://review.couchbase.org/114322 Tested-by: <ankit.prabhu@couchbase.com> Reviewed-by: Suraj Naik <suraj.naik@couchbase.com> * Commit: 6a8aeb18569928320265f71c4f449a37264ddd0d in build: 6.5.0-4230 MB-33126: Log unacked bytes during updater timeouts * Ack threshold: maximum number of unacked bytes. When unacked bytes exceed this limit view-engine will send the ack to memcached Change-Id: Iab50d888f738f73fdd95312e11ba1089639af8c1 Reviewed-on: http://review.couchbase.org/112885 Tested-by: <ankit.prabhu@couchbase.com> Reviewed-by: Suraj Naik <suraj.naik@couchbase.com> CHANGELOG for eventing * Commit: e1e821644199ed3935d6ca1b05cdb495730a919c in build: 6.5.0-4270 Merge branch 'unstable' PASS rebalance,functional,integration,n1ql,curl. See 2019-23.10.pass.html" target="_blank" rel="nofollow">https://ci-eventing.northscale.in/eventing-11.09.2019-23.10.pass.html Change-Id: I36d9117ba09d53b49fc84852a335bcb4b31ad80b * Commit: 6d8996415255294e7c7345203035e60c8a2dac9b in build: 6.5.0-4270 MB-30772: Ensure no timer is lost when v8 eventing process is killed Enable v8 workers to manipulate timers in C++ the same way as Go code can manipulate the bucket structure, so that the handoff from Go to C is persistent until C has finished firing the handler. Change-Id: I848a6b0cfc38e74edfa09332431608c3caf0dafc Reviewed-on: http://review.couchbase.org/109094 Reviewed-by: CI Bot Reviewed-by: <satya.nand@couchbase.com> Tested-by: <satya.nand@couchbase.com> * Commit: e5180a53ce8d47eab5039ea1a0bd7d6235b577d8 in build: 6.5.0-4263 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci-eventing.northscale.in/eventing-10.09.2019-21.01.pass.html Change-Id: Ia186b99c700395c31982949d2be6a7efb2efe3a4 * Commit: 0cd59b3afc830205585c6bcdf08ada7064c8fc32 in build: 6.5.0-4263 [MB-35886] Turn on is_executing_ in debug mode * The is_executing_ flag needs to be set before calling OnUpdate or OnDelete. Change-Id: I07c3ad6b64ddb8ded55503612024aa951da9268b Reviewed-on: http://review.couchbase.org/114581 Tested-by: Gautham B A <gautham.bangalore@gmail.com> Reviewed-by: Jeelan Basha Poola <jeelan.poola@couchbase.com> * Commit: c047f90dbe8dcae640fa37edcb5e2cc74b107878 in build: 6.5.0-4263 MB-35793: Bucket access disabled when function deployed. Bucket access disabled based on whether the function is paused or deployed. Change-Id: Iaafe32fe6084f24c9e87ecf2f11a8a80814aba60 Reviewed-on: http://review.couchbase.org/114508 Reviewed-by: Gautham B A <gautham.bangalore@gmail.com> Tested-by: Suraj Naik <suraj.naik@couchbase.com> * Commit: fe6b1bdc3370a91e07d6551c18fe2657e9f6149e in build: 6.5.0-4256 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci-eventing.northscale.in/eventing-10.09.2019-21.01.pass.html Change-Id: Ib7b6b4a3798661142f751b152a1b220fe11cef07 * Commit: 38f0c413903ef4911797bc708d2d9826e0a3cb2c in build: 6.5.0-4256 Upgrade changes from older versions to MadHatter MB-35742,MB-35739,MB-35817,MB-35842 This patch addresses the issues encountered during upgrade from older version of Couchbase Server to Couchbase Server 6.5. * The new REST apis added in MadHatter won't be called unless the entire cluster is made up of MadHatter nodes or above. * Rebalance issues during upgrade is fixed which was caused by flatbuffers schema changes. * Curl and bucket access is moved to app_config table in the flatbuffer schema to support forward compatibility. * Mutation not processing after upgrade from older eventing versions to new is fixed which was caused by bucket access field being null resulting in vector out of bound access in v8_worker.cc * Life cycle operation such as deploying and undeploying failing after upgrade is fixed which was caused by compressed handlers introduced in madhatter. The handlers are not compressed in case of mixed mode clusters with any eventing node older than 6.5. Change-Id: Ia2ed96896ec68ae9f93026b2ef73d0d2337b0f42 Reviewed-on: http://review.couchbase.org/114053 Reviewed-by: Gautham B A <gautham.bangalore@gmail.com> Reviewed-by: CI Bot Tested-by: Suraj Naik <suraj.naik@couchbase.com> * Commit: 1e749119bbff1d7ec05ef1f7b6cad5d481f274ff in build: 6.5.0-4256 MB-100: Cleanup build script Now build both cc and go files incrementally using build.sh and address a few warnings and remove dead code Change-Id: I9e2fdda01ec3133f16531eeb5add4c4342902299 Reviewed-on: http://review.couchbase.org/114441 Reviewed-by: Sriram Melkote <siri@couchbase.com> Tested-by: Sriram Melkote <siri@couchbase.com> * Commit: bd725526ae0b215d6c0e30faadf55349cc1aedb1 in build: 6.5.0-4233 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci-eventing.northscale.in/eventing-08.09.2019-20.29.pass.html Change-Id: I005dac0ffefa94440e412e5f6e244675580ba282 * Commit: a24d88f6453085cc090cb9818fee82840933be4a in build: 6.5.0-4233 [MB-35357] Ensure v8worker responds to control messages * Parsing metadata messages was done using v8. This required holding a lock on the isolate using v8::Locker. This meant that the parsing of control messages and processing of events could not be interleaved. We are now using nlohmann::json and thereby cutting down the dependency on v8. * There is a lock shared by V8Worker and AppWorker for updating/reading sequence numbers. The period of usage of this lock is now minimized in V8Worker. Change-Id: Iceed39891a7ee9a0fda672b59c3c7ceee5720736 Reviewed-on: http://review.couchbase.org/114383 Reviewed-by: Gautham B A <gautham.bangalore@gmail.com> Tested-by: Gautham B A <gautham.bangalore@gmail.com> CHANGELOG for eventing-ee * Commit: 8ac8ac0e103bb796fb71c8f3c15e8c42548351eb in build: 6.5.0-4270 Merge branch 'unstable' PASS rebalance,functional,integration,n1ql,curl. See 2019-23.10.pass.html" target="_blank" rel="nofollow">https://ci-eventing.northscale.in/eventing-11.09.2019-23.10.pass.html Change-Id: Id09e23add1904fb9d6108f19f234d77e1ebd2421 * Commit: 60d8b0be0d16c53d85834cb027ef1901ea6dc070 in build: 6.5.0-4270 MB-30772: Ensure no timer is lost when v8 eventing process is killed Enable v8 workers to manipulate timers in C++ the same way as Go code can manipulate the bucket structure, so that the handoff from Go to C is persistent until C has finished firing the handler. Change-Id: I1241d20de98c687198130aaddb1d13a0d62a181b Reviewed-on: http://review.couchbase.org/114537 Reviewed-by: CI Bot Reviewed-by: <satya.nand@couchbase.com> Tested-by: <satya.nand@couchbase.com> * Commit: 889ec8fe218f7a427fb73aa2a2e0c723a7dd0efc in build: 6.5.0-4244 Merge branch 'unstable' Change-Id: I0c7a415dda907da1de5486a8a4f3a7eaba59fefb * Commit: b19ff3c9de5bd9333ffeed127227d220ff968991 in build: 6.5.0-4244 [MB-35874] Exit js-evaluator when parent dies * Starting a thread to listen to in-pipe and call std::exit upon receiving EOF. * Adding .clang_format to keep the C++ code format consistent. Change-Id: Iae141743393aa96bf9f9ddfba13e4e8f650411f3 Reviewed-on: http://review.couchbase.org/114515 Reviewed-by: Gautham B A <gautham.bangalore@gmail.com> Tested-by: Gautham B A <gautham.bangalore@gmail.com> CHANGELOG for go-couchbase * Commit: 3db08c407d171d3bee5d86d1b33861d3c2ac27d2 in build: 6.5.0-4280 MB-35926: Handle port mapping when hostname is missing ... ... from the nodeServices endpoint. When the hostname is missing from the nodeServices endpoint, it means that the Couchbase Server node has been configured on the loopback address (127.0.0.1 or [::1]). This further means that the whole cluster must be configurd on the loopback (as nodes on other hosts wouldn't be able to communicate with a loopback-configured node since they wouldn't have its address.) So, if a client can reach a server port via a certain hostname and the node is configured on the loopback, it's safe to assume that the client is accessing the nodes via that hostname and it's sufficient to compare ports to map. Change-Id: I4226eebd82ed1bf51a270a5e12d114cb9e366178 Reviewed-on: http://review.couchbase.org/114713 Tested-by: Dave Finlay <dave.finlay@couchbase.com> Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> CHANGELOG for godbc * Commit: 818a303525ef1b19cdb2976c112ddca73f806d9f in build: 6.5.0-4274 MB-35900 revise alternate address for external network cbq shell connected to 8091 port netwrokcfg "external" look the query nodes in AlternativeAddress networkcfg "default" look the query nodes in internal address networkcfg "auto" given host is in Alternative use external else default once quernodes decided cbq shell will do roundrobin querynodes. Change-Id: If720ad4441c2f405c0c9a639bd8a237d47828ecc Reviewed-on: http://review.couchbase.org/114643 Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> Reviewed-by: Marco Greco <marco.greco@couchbase.com> Tested-by: Chang Liu <chang.liu@couchbase.com> CHANGELOG for gomemcached * Commit: 4e485a3d895fdb2a276c969e90fe32e1e8fa0280 in build: 6.5.0-4247 MB-35806 - More graceful error handling - Implemented version 1 with MaxTTL - Separate name getters between scope and collections - Updated unit test and data to better test getters Change-Id: I864061b0f0bcdd12119ed0e8045053b5ed733825 * Commit: 2508c27530e0b3b483a65a7fb58f58cf8c4037e4 in build: 6.5.0-4237 Merge "MB-35806 - gomemcached to handle collections system events" * Commit: 93520e9f1c2aaaef5a448bcf69374c9050973491 in build: 6.5.0-4237 MB-35806 - gomemcached to handle collections system events Change-Id: I8035da5fcdfe44062164a12ce96bfab4b9b74091 CHANGELOG for goxdcr * Commit: 49c07faa81b67da74b5c762d47715b645df4778f in build: 6.5.0-4266 MB-34568 - enable running cbworkloadgen on collection Change-Id: I07ee6e6b60d72f667a73b3ab31033fda2069b454 Reviewed-on: http://review.couchbase.org/114638 Reviewed-by: Lilei Chen <lilei.chen@couchbase.com> Reviewed-by: Neil Huang <neil.huang@couchbase.com> Tested-by: Neil Huang <neil.huang@couchbase.com> * Commit: 1aae8a31ae5ec62042287c1a324bf99c012b2899 in build: 6.5.0-4235 TXNJ-107 - Change filtering criteria to match new transaction naming convention Change-Id: I9113abaa7b3cd137e97287cee0eb06c882b779dd Reviewed-on: http://review.couchbase.org/114187 Reviewed-by: John Liang <johliang00@yahoo.com> Reviewed-by: Lilei Chen <lilei.chen@couchbase.com> Reviewed-by: Neil Huang <neil.huang@couchbase.com> Tested-by: Neil Huang <neil.huang@couchbase.com> CHANGELOG for indexing * Commit: 2abc0fd0476345d6069e55f2cecbb316cd261827 in build: 6.5.0-4281 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci2i-unstable.northscale.in/gsi-13.09.2019-05.30.pass.html Change-Id: I1cb79e581e5dbae6307eeb7404730d6d0f3125dc * Commit: e29ca4ed37e81155bec62b54a1c870552a76d7f3 in build: 6.5.0-4281 MB-35880 mark disk snap as used only for rollback to zero Change-Id: I124d18a7cab45f4d0feaa3a66664aba83d281704 * Commit: b89cee733ce78507711c083e874ca0edc7d02fc8 in build: 6.5.0-4281 MB-35880 fix restart timestamp calculation for rollback during rollback, the disk snapshots across indexes/partitions could be ahead/behind of each other due to: a. disk snapshots may happen at different times for different indexes or partitions as it is async. b. during stream merge, the snapshot from INIT stream could be ahead of snapshot in MAINT stream. Due to this, it is important to calculate the minimum common timestamp across indexes/partitons which could be used to restart the stream. Change-Id: Ie9445972014f76ce8ded7eb74495c9bf55ad0d6b * Commit: db4374a0579817c5fc3ad3ef5f76658bb4e87a9e in build: 6.5.0-4281 MB-35880 fix restartTs calculation after warmup the current AsRecent check compares vbuuid as well. The problem is that due to vbuuid mismatch, a higher seqno timestamp could be chosen as restartTs resulting in missing mutations after warmup. vbuuid could mismatch in non data loss scenarios due to memcached crash and restart. Change-Id: I85b6a3624c7749d5e124bf8cb12ce9afc41a5eb1 * Commit: 5e9dc00479f62ba2a33cd02eb7ebd42541c41a7e in build: 6.5.0-4281 MB-35545: Improve overall error handling in TransformRoute In case of panic recovery or error in TransformRoute, skip the culprit document. Send upsert delete to the indexer for the skipped documents. Maintain IndexEvaluator stat for number of items skipped. Report the skipped document count in console logs. Change-Id: Id177a4fea4f7cd6be6d4ad78e83b5550ae818901 * Commit: 30f619d0d95b2f729ffe2d55dc0808e4f3f88104 in build: 6.5.0-4246 Merge remote-tracking branch 'couchbase/unstable' into HEAD http://ci2i-unstable.northscale.in/gsi-10.09.2019-14.16.pass.html Change-Id: Iad80b8572915b2abaccd1ab53b6647489a5ae947 * Commit: cc9bde1da33b291a7e0dfd1984cf6340af357b69 in build: 6.5.0-4246 MB-31787 Fix diskSize reporting a. Use GetLSSUsedSpace() instead of GetLSSInfo() b. Add checkpoint file size to disk size Change-Id: I20d2eded2f650c23b0024262e5381a8198c4d914 * Commit: 5e8e0fcb1f33fee5741e158ac9eac8422a02a218 in build: 6.5.0-4246 MB-35699 Use hexadecimal opaque when logging projector stats The code initially contained hexadecimal representation of opaque value. When stats_manager was introduced, integer representation was used at some places. Using hexadecimal representation throughout to prevent any confusion Change-Id: I547cbd42c45e1129ef65f0036b648d625148976c * Commit: ed52911edf60312d0346849dfd8dd8242914c61d in build: 6.5.0-4246 MB-34937 Do not log Fatal too early in the bootstrap Change-Id: Ifb72bc9b64ca71a23b963b8178e011e848fa73e1 * Commit: 9b07e2ca6f820622a6a37004509fb5928f255af4 in build: 6.5.0-4246 MB-35642: Add plasma setting for evicting dirty page Change-Id: I367dfbd821e6772471c8b14d24ca42bca948c918 * Commit: 8a803ef55e9c7fa4c1d20525f244ae8c9ea6c416 in build: 6.5.0-4246 MB-35803 fix fast count for desc field inclusion gets flipped for desc field Change-Id: Icef9fa438ad3d915d01b7be8c1339792502798e1 * Commit: 589fc9ba72ab1451d16a96cc65efee7c19edef82 in build: 6.5.0-4246 MB-35603: increase max page size Change-Id: I0abdd471170260c847954a346d1bf435bc27d5cd CHANGELOG for kv_engine * Commit: a078232c971bb2f493f7468eaade9f059d6659df in build: 6.5.0-4277 Clean up EPE::getStats() * Move larger code blocks to separate methods to handle the requested stat block. * Use the cb::const_char_buffer instead of copy it to a std::string * Use early return instead of else if Change-Id: I8c7d0de42bc452bc5a8b49b3fa9a0e3ee9bfcfce Reviewed-on: http://review.couchbase.org/114594 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: 413f71d4ebf09c26f6e4574833d780982ad20298 in build: 6.5.0-4277 MB-35930: Add support for providing stat args Allow the STAT call to contain a value containing a JSON payload containing "arguments and options" to the given STAT call. It is up to the stat call to define the schema for the JSON provided to each stat group. Change-Id: I8e88cddbcb8e7c9ceea22e368f5cc71238618f20 Reviewed-on: http://review.couchbase.org/114585 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: 3a6d7505f415176654c0c0c16542451e036521a2 in build: 6.5.0-4272 MB-35938: Add support for "unselect bucket" Clients may go back to the no bucket by selecting "@no bucket@" (which is an invalid bucket name so it may never be a conflict with a real bucket) Change-Id: Id83d8a7bb1968cdc9684b64b5088542c64133db5 Reviewed-on: http://review.couchbase.org/114670 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: 324d9cb59a17a8cfec746836cfc929b434f05416 in build: 6.5.0-4272 MB-35933: Ensure duplicate prepare replaces correct prepare In cases where a prepare is allowed to replace an older prepare (after a disconnect/reconnect, if the intervening commit/abort may have been deduped), ensure completed prepares are not considered for replacement. Completed prepares may not be immediately removed from tracked writes if they are PersistToMajority, as they must be kept until the prepare is persisted. A duplicating prepare will find the existing prepare in the hashtable, and is only permitted to replace *that* prepare in trackedWrites. Prior to this change, an `Expects` would fail if an older completed prepare is encountered first. This patch ignores completed prepares when searching for the correct SyncWrite to replace. Change-Id: I85bc660441a6036a53218b5f42a53229bc6dc8ae Reviewed-on: http://review.couchbase.org/114615 Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 9a7b435eb0b88b277138458d6670125857bf21e2 in build: 6.5.0-4271 MB-35934: Skip deleted vBuckets in DurabilityCompletionTask The task incorrectly assumed that if it was scheduled for a given vBucket then the vBucket would still exist. Skip these vBuckets if we cannot acquire the shared_ptr to them. Also, fix an issue where we incorrectly did not wake the task on the first SyncWrite that should schedule a vBucket (sets pendingVBs[vbid] from false to true). Previously we would only wake the task for any other transition (e.g. true to true). Change-Id: I176e68b9f605408b5f704023c30847ad56ebd79d Reviewed-on: http://review.couchbase.org/114619 Reviewed-by: James Harrison <james.harrison@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: 9c27fb91f40ff587a43b5768d887f2c3cabf66a5 in build: 6.5.0-4264 MB-35932: Distinguish a bg-fetch needed vs pending durability The KVBucket set/add/replace/deleteItem methods were overloading the meaning of EWOULDBLOCK for pending sync-writes, in EWOULDBLOCK was returned for sync-write accepted and bg-fetch needed. The caller of the method then assumed a pending set for example returning EWOULDBLOCK was always a sync-write accepted, breaking SET with CAS in full-eviction (and others). To address this issue, add a new error code ENGINE_SYNC_WRITE_PENDING so that the caller of KVBucket methods can distinguish sync-write accepted vs needs bg-fetch. Change-Id: I91cc4883dc15c145ff9293ad680beb9063028e66 Reviewed-on: http://review.couchbase.org/114596 Reviewed-by: Trond Norbye <trond.norbye@couchbase.com> Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: f9fd16f1b4d2cc37ef96da4f4d760fbbfb537c2c in build: 6.5.0-4262 Flip the logic for Reorder bit to Barrier Clients want to be able to set "unordered executions" as the new default, and they don't provide any guarantees about execution order today (a change in the cluster topology could already reorder the commands for the same key) If the client enables unordered execution for a connection, the server is permitted to choose the execution order for all commmands on the connection (except for commands with a barrier bit set). Change-Id: I26f3af7e831333ff8c1b41e1d3e45b3137ae0106 Reviewed-on: http://review.couchbase.org/114329 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: d58119c41c8c8036bdd7571d97d4aec594101de0 in build: 6.5.0-4261 MB-35928: Disable SASL auth for DCP connections SASL auth could result in switching buckets. Change-Id: I9a1ecd51ace79953fdfe705a1ac7b6fee47b887b Reviewed-on: http://review.couchbase.org/114584 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: e3ccc2835f90270ae47c0e2466c967d5471b17ba in build: 6.5.0-4261 MB-35904: Disable select bucket for DCP connections The connection object have running tasks in the underlying engine, and the engine may keep references to the connection object. If the connection switch bucket it'll try to call back into a different bucket and things will go terribly wrong. (Note that DCP connections require a special privilege) Change-Id: I195288f15404bfd565ca7ac5dbc43854c0e80e80 Reviewed-on: http://review.couchbase.org/114559 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: James Harrison <james.harrison@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: 69044aeef5ef670fee9a3b74d739fd03e304990e in build: 6.5.0-4257 MB-35631: Do not skip checkpoint_end ops for persistence cursor The flusher relies upon checkpoint_end ops to know a disk checkpoint has been completed and the associated highCompletedSeqno can be persisted (on a replica). Change-Id: I240a09436b592cf2e51bd4c507575d5df9d1a7f2 Reviewed-on: http://review.couchbase.org/114440 Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: c9d66dc90ee8878ba5df8ff7b44244a001c99f65 in build: 6.5.0-4247 Remove Ephemeral DCPBackfillMemory (non-buffered) Remove the DCPBackfillMemory class, leaving only the DCPBackfillMemoryBuffered sibling. During Ephemeral develoment we initially had an in-memory backfill class (DCPBackfillMemory) which performed backfill "atomically" - lock the seqList, take a copy all items in the given range, then unlock. However this resulted in (a) potential slowdown for front-end ops whils seqlist was locked and (b) large spikes in memory to handle the large number of items being moved around. An alternative sibling class - DCPBackfillMemoryBuffered - was added which makes use of the range read iterator support to allow iteration over the seqlist; while still allowing concurrent access from the frontend. This class has been used by default since Ephemeral was released. Remove the essentially dead non-buffered variant - we've never had cause to switch back to it, and it just adds extra maitenance to keep it. Change-Id: I406862cfe31b09acf3f4bc3e06484d6d27e91e76 Reviewed-on: http://review.couchbase.org/114071 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: James Harrison <james.harrison@couchbase.com> Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> * Commit: 3e25fedd22724dcf0e471489d3f0c0a3f61f3eff in build: 6.5.0-4246 MB-35631: Only persist HCS after complete disk snapshot persisted Currently, the HCS sent as part of a disk snapshot marker will be persisted the first time any items from the checkpoint are flushed to disk. This lead to non-monotonicity as the flusher would later try to persist a /lower/ HCS upon encountering a commit. This patch ensures the disk snapshot HCS will only be persisted if the current flush batch reaches the end of the disk checkpoint. Change-Id: I4fc6326c6c4db814a43ebe6f552d9d3bc96b5448 Reviewed-on: http://review.couchbase.org/114245 Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: f123169ccf663588791ff930611bb7f4d9ce0bde in build: 6.5.0-4245 MB-33463: Implement logical delete for magma Add new kvstoreRev parameter to track kvstore revision. It's maintained on magmaInfo which is updated as part of vbstate. Implement prepareToCreateImpl to increment the kvstoreRev. Implement prepareToDeleteImpl to soft delete the kvstore in magma. If the vbucket exists, it must call prepareToDelete prior to calling prepareToCreate or exception will be thrown. Magma SoftDeleteKVStore effectively deletes the kvstore by moving the shared pointer to the kvstore to a pending delete list. Subsequent attempts to access that kvstore in magma will return NotExists status. The kvstore storage should eventually be deleted when delVBucket is called. It is at this point we wait for all users of the kvstore to complete any outstanding operations before removing the storage to the kvstore. In the event of an unexpected outage where there are dropped kvstores still pending, recovery will delete old kvstores up to the latest kvstore. Change-Id: Ida4555663c31f79cca53b7ff86270527f49f01ae Reviewed-on: http://review.couchbase.org/113850 Reviewed-by: Jim Walker <jim@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 85ac9e66e7facf6baafd930476b82f0002ba4733 in build: 6.5.0-4245 MB-35764: Do not try to close checkpoint on snapEnd Prior to this patch on receiving a snapshotEnd, a PassiveStream can attempt to close the open checkpoint if mem_used is above a threshold. The original intention of this was to try to free memory tied up in the checkpoint by closing it and allowing it to be deleted when unreffed. Recently checkpoint expelling was added which can recover memory from a checkpoint by freeing individual items without requiring the entire checkpoint to be deleted. With expelling, closing a checkpoint in this situation is no longer necessary and is removed with this patch. Now checkpoints on a replica will only be closed when instructed by the active (snapshot with the CHK flag). Change-Id: I30a24a9293b4b63c4940514ec20c792751eaba3c Reviewed-on: http://review.couchbase.org/114117 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> * Commit: ff3c2e623e8b99a8258d2004a8dcdd727f48e66c in build: 6.5.0-4244 MB-35812: Ensure correct cursor is selected when expelling from ckpt Checkpoint expelling expects that the "earliest" cursor will reside in the oldest referenced checkpoint. While this is correct, "earliest" was previously defined as the cursor pointing to the item with the lowest seqno. However, the same seqno can in certain situations appear in two checkpoints (if a checkpoint ends with a set vb state, see CheckpointManager::createCheckpointItem for the logic which sets the seqno for the meta item). Given this, the ordering used to determine the earliest cursor is insufficient, and could select the later cursor. To resolve this, find the cursor with the lowest seqno and checkpointId. Change-Id: Ie28279a4a21fb6e9804bca0f370950ed135e97cb Reviewed-on: http://review.couchbase.org/114376 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> * Commit: cf93325a999f1ae117c07d49385bc166ebced649 in build: 6.5.0-4244 Revert "MB-33423: Temporarily replace exception with abort" This reverts commit f254f253558f4b549d2bc964d940f9d3a192d0a5. Change-Id: Id0cf1a6836d177fb0f9d0b33f54b253c5d7b435c Reviewed-on: http://review.couchbase.org/114239 Reviewed-by: Trond Norbye <trond.norbye@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: fb36bc9645f709f5aad05f7355ab4a13517fce76 in build: 6.5.0-4242 MB-31450: Track max-deleted revision using checkpoint::queueDirty When a checkpoint encounters a DEL(k1) then SET(k1) it will discard the DEL(k1) (de-duplication). A consequence of this is that the flusher will not be able to update the vbucket meta-data (vbstate) with the correct "max-deleted revision seqno". As shown in the test this could mean that conflict resolution could incorrectly allow a mutation because a warmup occurred. To address this issue, EPBucket::flushVBucket no longer inspects the items looking for deletes, instead Checkpoint:;queueDirty does the inspection before de-duplication occurs. Each Checkpoint now has an optional member which when initialised, stores the largest revision seqno of all queued deletes. The flusher now gets the value to persist from the checkpoint-manager. Change-Id: I5dfa7a4cb1e0359eef83b7e1df84f9448c57c43c Reviewed-on: http://review.couchbase.org/114442 Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: d3e6d639696c09329d594dd62e2761f408dc59fa in build: 6.5.0-4242 MB-35061: Add test confirming Consumer correctly removed from vbConns Change-Id: I97a2fd99bbbf6325e4ad8e8392a3dcb64e5c7a9c Reviewed-on: http://review.couchbase.org/112838 Reviewed-by: Ben Huddleston <ben.huddleston@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: c29077cd010be0a74158df8df893afa5e87b54c2 in build: 6.5.0-4232 MB-35592: Fix hang situation in cluster_test The test for MB34780 tries to shut down the replicator when it sees a DCP_PREPARE being transmitted over the DCP stream. The problem was however that it started off by deleting the default bucket and then recreate the default bucket _WITHOUT_ deleting the database files first. That means that the second time it created the database it would actually "recover" the old database and a prepare could be resent (and depending on the timing we might not get all of the dcp streams set up before we started to drop packets) Change-Id: Ia8719618655330876c2430bc42ca9cd6164a04fd Reviewed-on: http://review.couchbase.org/114439 Reviewed-by: Jim Walker <jim@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: f268a1d3b5f98b1250b0f2153f71c5a4e49eb1d5 in build: 6.5.0-4222 Split StreamState into an enum for each Stream subclass Split Stream::StreamState into 3 individual enums for each of the three Stream subclasses - ActiveStream, PassiveStream & NotifierStream. This makes all the state transitions explicit (no invalid states are possible for each class). Change-Id: Id4643019526119c6dd2420165da7b44a3014d497 Reviewed-on: http://review.couchbase.org/114340 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: James Harrison <james.harrison@couchbase.com> * Commit: 297e39b682a320f9ee322b3a17e16da939a9fbee in build: 6.5.0-4221 [cluster_testapp] Let the bufferevent close its sockets We had an explicit close of the socket and used unique_ptrs for the bufferevent structure. Calling bufferevent_free will remove the listening mode for the socket, but if the socket is closed we're trying to operate on an "invalid" socket. Given that there is an option to bufferevent to close it when its done with it lets use that so that we don't have to care about the ordering Change-Id: I36f29f04163e2024590e34418a177c034fb5bff4 Reviewed-on: http://review.couchbase.org/114370 Tested-by: Trond Norbye <trond.norbye@couchbase.com> Reviewed-by: Jim Walker <jim@couchbase.com> CHANGELOG for libcouchbase * Commit: 9a3fbfd331d5ddd208c3cf2a3779c589c3917dd7 in build: 6.5.0-4272 Make ciphers and minimum TLS version tunable Allow the user to specify the minimum TLS version to use and which ciphers to use through the environment variables. TLS is set through LCB_SSL_MINIMUM_TLS which may be set to: tlsv1.1 tlsv1.2 tlsv1.3 If set to an invalid/unsupported value the minimum version is set to TLS1. Unfortunately OpenSSL use two different API calls in order to specify the ciphers to allow for the various TLS levels, and the methods work sligtly different so it has to be tuned through two different environment variables. Ciphers for TLS < 1.3 is set through LCB_SSL_CIPHER_LIST and for TLS >= 1.3 it is set through LCB_SSL_CIPHERSUITES (the names in use match what OpenSSL calls them). Two new error codes was added to the system: LCB_SSL_INVALID_CIPHERSUITES - We failed to set the ciphersuite LCB_SSL_NO_CIPHERS - OpenSSL was not able to select a single cipher from the list provided. Change-Id: I48be0d943a360ec1008e17535b791bc2735c2f26 Reviewed-on: http://review.couchbase.org/114677 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Sergey Avseyev <sergey.avseyev@gmail.com> CHANGELOG for magma * Commit: 4816624f646018af34cc3996f2ea968fb8fd3263 in build: 6.5.0-4280 Revert "magma: Use logical size for compaction only for level above LSD level" Reverting commit since it results in very large sstables containing LSD delete deltas. This causes massive compactions instead of lots of tiny one. This reverts commit bdfaf3abc12bf9ea9de8675129f59d286fe9371a. Change-Id: Id7de94774f3c9e088d148d86dde61e39392bcf6e Reviewed-on: http://review.couchbase.org/114672 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Scott Lashley <scott.lashley@couchbase.com> * Commit: 1f27db555cf590b9a209b4bd8ba9b3a2f4dcf9d7 in build: 6.5.0-4280 Revert "magma: Add LSDMaxCompactSize to magma config" This reverts commit e2a035552e82fa283a155c352b80ffc8250f8949. Change-Id: I9e3a7157ceabde94069d587aca00d7d71ac16360 Reviewed-on: http://review.couchbase.org/114671 Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Scott Lashley <scott.lashley@couchbase.com> * Commit: 816aa1f9e2c4f4a7a6732491bac1583ba84f9f74 in build: 6.5.0-4243 CBSS-267 lsm: Add block type to block pointer Change-Id: I4bf7ebebfb4d5249f3b928f3e41b8d47f37798c1 Reviewed-on: http://review.couchbase.org/114510 Reviewed-by: Sarath Lakshman <sarath@couchbase.com> Tested-by: Build Bot <build@couchbase.com> * Commit: 5a4abc09f5df3a6b1248ef17644cf62fddc2f075 in build: 6.5.0-4235 util: slice Add char* constructor for Slice Add new constructor for slice to take char* only and calculate the length using strlen(). Change-Id: I85a5a314d376b8b42bc7cb0eb86dc5dea60955d3 Reviewed-on: http://review.couchbase.org/114456 Reviewed-by: Sarath Lakshman <sarath@couchbase.com> Tested-by: Scott Lashley <scott.lashley@couchbase.com> CHANGELOG for n1fty * Commit: 259e6a3a666d155f65da9c9e1e8f9553e6541ffe in build: 6.5.0-4250 StreamSearchResults_SearchResult doesn't carry any hits + total_hits does not represent the actual hits sent within the search result, the hits will be streamed as StreamSearchResults_Hits. + Set numHits to 0 in this case, as backfill doesn't need to be set up for this scenario. Change-Id: I6e8798ba365ad172341d839a20566de106f2aca8 Reviewed-on: http://review.couchbase.org/114563 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Abhinav Dangeti <abhinav@couchbase.com> Tested-by: Abhinav Dangeti <abhinav@couchbase.com> * Commit: ea648270c4b49bc2785bb697d80a822188630aed in build: 6.5.0-4248 Determine number of hits correctly for backfill decision Change-Id: Iad9d55b01de5adbdebb9bcefe8dff302a4cda391 Reviewed-on: http://review.couchbase.org/114493 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Yen <steve.yen@gmail.com> Tested-by: Abhinav Dangeti <abhinav@couchbase.com> * Commit: ca6f6c48571ea2adb68c9ae1678d6180c542b982 in build: 6.5.0-4239 MB-35424: Do not reset opaque while checking Sargability Change-Id: Id2ff85fedeea71d8df933ad2747c77b12a70e43f Reviewed-on: http://review.couchbase.org/114497 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Abhinav Dangeti <abhinav@couchbase.com> Tested-by: Abhinav Dangeti <abhinav@couchbase.com> * Commit: 307c8084fd316696a8add3068fd076877847162a in build: 6.5.0-4239 MB-35424: Sargability: check if query_fields are already available Change-Id: I74b822ecbf9ad06b97f536e0664171d069cf675d Reviewed-on: http://review.couchbase.org/114403 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Yen <steve.yen@gmail.com> Tested-by: Abhinav Dangeti <abhinav@couchbase.com> * Commit: 7915809b898cf81b879ac4a59880cc9f5eb7df22 in build: 6.5.0-4235 MB-35424: Allow user to skip verification of search results The user can now choose to provide within "option" a setting that will allow N1FTY to skip evaluation. Here is the setting: "skipVerify": true -or- "skipVerify": "true" Change-Id: I7cac1021c1ac41636ceb0d23c19392367e651978 Reviewed-on: http://review.couchbase.org/114350 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Yen <steve.yen@gmail.com> Tested-by: Abhinav Dangeti <abhinav@couchbase.com> CHANGELOG for ns_server * Commit: 281b0233e52ee6333fd04fe722adb0ee2276ffcd in build: 6.5.0-4268 revise focus chart layout & language Change-Id: Id26560b508b748d3207281d0765221dbe0ebd1c1 Reviewed-on: http://review.couchbase.org/114644 Reviewed-by: Rob Ashcom <rob.ashcom@gmail.com> Tested-by: Rob Ashcom <rob.ashcom@gmail.com> * Commit: c12bb8400d19d184854c6d1ae5e1f0ddbac4a980 in build: 6.5.0-4254 MB-34598 Fix accidental vbucket deletions due to node rename. The issue may occur if node gets renamed while a janitor run is in progress. The sequence of events is as follows: - ns_janitor gets a bucket config when the node name is node1. - node1 is renamed to node2. - ns_janitor sends the apply_new_config call to janitor_agent. - janitor_agent checks the vbucket map and finds that no vbuckets should reside on node2, since the vbucket map still uses the old name. - janitor_agent proceeds to delete all vbuckets. We tried addressing this by stopping most of ns_server processes during node rename. But that proved harder that we'd have hoped and also introduces service unavailability. So this patch instead works the issue around by detecting that node rename might have happened when handling apply_new_config and apply_new_config_replicas_phase calls in janitor_agent. If it detects that rename occurred, it will refuse to handle the call. Change-Id: Ia64681a60d24f8aa3428f13f39149c15b4e57725 Reviewed-on: http://review.couchbase.org/114573 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: a4914cb0a688afb3f2a839f19723f14e03b3850c in build: 6.5.0-4254 Move apply_new_config{,_replicas_phase} calls into subfunctions. Change-Id: I1e4467e359e9bee9f886f095184ef75b62f2064b Reviewed-on: http://review.couchbase.org/114572 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 3ebb6d17cfd7dbd2109b8446114a21def9a15c4c in build: 6.5.0-4254 Address a TODO. Change-Id: I49b21bb33bd01a92c818c5005826de20d2c0776b Reviewed-on: http://review.couchbase.org/114571 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: b0f8a1b56374fa1769670586882a3371f8d30b41 in build: 6.5.0-4254 Don't wait for auto_rebalance process when casting. casts are by definition asynchronous, and in this particular case are only used by ns_orchestrator which resides in the same supervisor which guarantees that both processes are alive. Change-Id: I4979a47ffa25b4e07218eb0ba6ca5a7b0a1691bf Reviewed-on: http://review.couchbase.org/114570 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 5214ff0711869fa99d55c9578d3a3db7c56b84fe in build: 6.5.0-4254 Remove rebalance retry cancellation from the REST API handler. ns_orchestrator will do this when the call actually reaches it. Change-Id: Ia514f0f3f119679fc3417261dc0766e84b3fb153 Reviewed-on: http://review.couchbase.org/114569 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: f782b64c0a93d6b188d4d919aeeea319c25dad42 in build: 6.5.0-4254 revise query monitor footer CSS for tiny charts Change-Id: I276b18f7780846b67e4348ac6a364771ebc32783 Reviewed-on: http://review.couchbase.org/114575 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Rob Ashcom <rob.ashcom@gmail.com> Tested-by: Rob Ashcom <rob.ashcom@gmail.com> * Commit: 0417b64ffeff21817791242d14f44620d74bf13d in build: 6.5.0-4252 Always terminate dependent async-s with reason 'shutdown'. If we're explicitly terminating dependent async-s due to some kind of error, it means that this error will be visible either because it's propagated to the user of the async or because a crash report will be generated by our async. Since the error is already visible, there's no point in propagating it further down the async tree. Change-Id: I73b48152a1e04db99e2e64e27d2f259f8f7eac64 Reviewed-on: http://review.couchbase.org/104297 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: bc547fc3a26b2e7bd85680182a7448d1ac841dc8 in build: 6.5.0-4252 Prevent a crash report when propagating exceptions in asyncs. Since we are propagating the exception, there's no need to produce a crash report in the async code: either the caller will die and hence produce the report, or it will handle the exception and log something if appropriate. Change-Id: Ia64b499e0903371a2dd5fe0ac3e022e15c4f96ce Reviewed-on: http://review.couchbase.org/104296 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 43a1dfcec9555ad51da6790d2be75df0c905ccd4 in build: 6.5.0-4252 Basic test ensuring that async-s propagate exceptions properly. Change-Id: Ic644d26eaac1aeec5c2983ed80d15f004b8b2f0d Reviewed-on: http://review.couchbase.org/104295 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 23913cadb036bbd07f4e503de8d721c007b65df2 in build: 6.5.0-4252 Don't produce crash reports in async due to monitored processes. The monitored process would have produced a crash report already. So we avoid polluting the logs with "derived" crash reports. Change-Id: Idd4a3a3a98d3440f43b5cdfa029c175089e1d424 Reviewed-on: http://review.couchbase.org/104294 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 2f96449b511f18e2d097778c7756c915bc3ec562 in build: 6.5.0-4252 Reimplement async:run_with_timeout using abort_after option. Which avoids spawning unnecessary processes. Change-Id: Ida90dfec7805eb44cd3c458e416097e7a021f94b Reviewed-on: http://review.couchbase.org/104293 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 792b74740276bfb6479684b940dd524c40da3752 in build: 6.5.0-4252 Get rid of async:perform/{1,2}. What it was doesn't fit very well into what async-s were intended for. So to simplify the code I decided to get rid of this misfeature. Change-Id: I6ef94fe3716e22e6426f3dd144d58df444514a0b Reviewed-on: http://review.couchbase.org/104292 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 6361275346b01661315c668e1e3a7e7ebdf53534 in build: 6.5.0-4252 Don't use async:perform/1 for gen_server2 asynchronous jobs. gen_server2 module cannot trap exits, since it's something that is decided by the callback module. So the job spawns a watcher process that monitors the parent and will kill the job if the parent dies. Change-Id: Ibd4e4d9bb243281cb71528726271d72b5b718ed9 Reviewed-on: http://review.couchbase.org/104291 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: a863f4c5b7c1ad9ec9f2b51a876d6548b7d3577f in build: 6.5.0-4252 Test that gen_server2 jobs terminate when the main process dies. Change-Id: I6f164b91ed7e5228fa154cedb01f6c7059ae692a Reviewed-on: http://review.couchbase.org/104289 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> * Commit: 946d3289e2463f8cf80f730c2ba65e3510019a24 in build: 6.5.0-4252 Add a simple test that exercises async:wait_any/2. Change-Id: I4a61638ad759ae57f5aed5727cba49ac3cf37646 Reviewed-on: http://review.couchbase.org/104286 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: c1ea0c22f649b9e93abedd075cf556bef5ec27d4 in build: 6.5.0-4252 Run eunit tests for each module in its own process. That way there's less chance for unwanted interactions between the tests. Examples of the interactions are: - Some tests don't bother to cleanup messages after themselves, while other tests assume that their mailboxes are empty when they start. - If a test forgets to kill a temporary well-known process, other tests might succeed if they depend on that well-known process but don't bother to set it up themselves. Change-Id: I09f00852a2705bbc338b0871f62b212e230c01ec Reviewed-on: http://review.couchbase.org/104284 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> Tested-by: Artem Stemkovski <artem@couchbase.com> * Commit: 7a2c1442a0e9b81231d325ee6296664288418a8a in build: 6.5.0-4252 Don't run tests defined in _tests modules twice. This happens because eunit:test(module) will also run tests defined in module_tests. This commit will filter out _tests modules if their corresponding main module is going to be tested anyway. Change-Id: If2eb262a6952ba72d13b60a3c72c41205c9d7797 Reviewed-on: http://review.couchbase.org/104283 Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> * Commit: 12019c27e95426d8b2c064d34b960e05e6bc2f72 in build: 6.5.0-4252 Fix menelaus_web_settings tests. Specifically, parse_post_data_test() and find_key_to_delete_test() implicitly (through a call to conf(security)) depended on ns_config being present. Yet the tests didn't arrange for ns_config to be running and only succeeded when run in conjunction with ns_config tests that left an ns_config instance behind. Change-Id: Ie43152446b25754027ccfc02707334fbc1365551 Reviewed-on: http://review.couchbase.org/114410 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: a95a95fe57b7202180f0e0eaacbce30c2b535b09 in build: 6.5.0-4252 Terminate cbcollect_info gracefully. Unfortunately, cbcollect_info can't be killed brutally. It creates temporary files that need to be cleaned up when it's terminated. If killed brutally, those temporary files won't get deleted and will continue occupying space in TMPDIR. This reverts the following commits: b5371e958ae4d65ab94b51ce927221ccada5c62d bc045e3111013018eba024cc0ba28b8c703bec96 Change-Id: Ifea36092314049645735ebb4c2eae698a11a81ff Reviewed-on: http://review.couchbase.org/114428 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> * Commit: 71725ca6cfd42b5024c8729a18d2869b78ec00e7 in build: 6.5.0-4252 Allow telling run_external_tool how to terminate the process. Change-Id: I4d6c88b87532fb5c8037ad1793df7252dd983090 Reviewed-on: http://review.couchbase.org/114427 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> * Commit: 33f7ff7afa3ef9f2fa01e3642b7a37cb0eeb3592 in build: 6.5.0-4252 [goport] Revive --graceful-shutdown option. It is needed, after all. Change-Id: I8a5436af0ae20ea02a27d4eeab81a8a2031b12b7 Reviewed-on: http://review.couchbase.org/114426 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: fe2174788c6651429595baedb4c409ff5f8cc71d in build: 6.5.0-4251 [goport] Make sure ProcessWorker responds to cancel requests. Even if the underlying process doesn't produce any data. In such case the worker will be blocked in the inner select that doesn't check the cancellation channel. The motivation in having nested selects was probably to give cancel requests explicit priority: if more than one select clause can be executed, the one to actually get executed gets picked at random. But besides a slight chance of not reacting to cancel requests immediately when there's data coming from the process, it's fine to have it all sit in a single select statement. Change-Id: I7234fdb7b25f8f7d12a0cacdd5e31c0fde7b7427 Reviewed-on: http://review.couchbase.org/114425 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 626828f4116abe359b31ce61e0dc2b044fdf7425 in build: 6.5.0-4250 Add support for "tiny" sized strip charts based on chart directive. Change-Id: I716ed52909e2e30ce2db99fe4aac9807ad3cbc2a Reviewed-on: http://review.couchbase.org/114178 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Rob Ashcom <rob.ashcom@gmail.com> Tested-by: Rob Ashcom <rob.ashcom@gmail.com> * Commit: 8006c45bcc42036b3efaad581cb441992950e178 in build: 6.5.0-4250 use ns_bucket:get_servers everywhere to get servers in some cases we used [] as a default, but it looks like we can safely assume that servers field is always present Change-Id: I6b5acbbd653c163acb65e64deffe2f8a15fb5af6 Reviewed-on: http://review.couchbase.org/114484 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Artem Stemkovski <artem@couchbase.com> Reviewed-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> * Commit: 5aef0789e6cebd5037fa53df2ffe99414134c987 in build: 6.5.0-4249 rename ns_bucket:bucket_nodes to get_servers ...so it does not have word 'bucket' in the name and consistent with ns_bucket:set_servers Change-Id: I7d9a1d7618498455c73e525d4beecc9ff1412467 Reviewed-on: http://review.couchbase.org/114483 Tested-by: Artem Stemkovski <artem@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> * Commit: 58711516501e0c9d4dfd76189a05ab8777cff179 in build: 6.5.0-4248 wip: add items dropdown selects Change-Id: Ia1c617f9650fc9745553f68a9308014939fa1add Reviewed-on: http://review.couchbase.org/114140 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Rob Ashcom <rob.ashcom@gmail.com> Tested-by: Rob Ashcom <rob.ashcom@gmail.com> * Commit: 69a1be6a14b8a5431074d95fdf63916938124e0d in build: 6.5.0-4239 [goport] Remove unnecessary empty line. Change-Id: Ia66422250c0257e1429d47c3000c6ce1b7509f41 Reviewed-on: http://review.couchbase.org/114424 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> * Commit: 7f4c7b39725fe22729784c72c1dbc18b9fb0ed7a in build: 6.5.0-4236 handle empty data.samples in chart directive Change-Id: I11ee0d868313091e5001b32a661a0af7aa97fc3b Reviewed-on: http://review.couchbase.org/114454 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: 5d42fb6101c7d758626fd4b0a34de588066e1194 in build: 6.5.0-4236 MB-35788: show message in case there is no such chart Change-Id: Ib1d5686c0ecd77383d7e87c9466eba8f2bd2bd80 Reviewed-on: http://review.couchbase.org/114417 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: 7dd26e069266dbc06712f8bfb54e74c058fdaec0 in build: 6.5.0-4234 added fixed height for chart container Change-Id: I317a5c74ffe152d49310bb1b79b1951801d0382b Reviewed-on: http://review.couchbase.org/114416 Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: cd46317ab79ff0b54ea5b2f40cba8d8c3514382b in build: 6.5.0-4234 MB-35801: reduce chart top margin Change-Id: Ic1ee580af010ef77426886b95b358396a73e1d32 Reviewed-on: http://review.couchbase.org/114415 Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: 0dc4158daa31525d60cb6fc8f9d53334d41f2707 in build: 6.5.0-4229 MB-35770 Return default TLS to unknown clients The cbindex utility uses the reverse rpc mechanism in order to reuse existing code in other services. When the service specific TLS support was implemented, cbindex wasn't included as a known service and thus ns_server didn't return TLS info. With this fix, for a "service" (such as cbindex) that isn't known we'll now return default TLS info. stevewatanabe @ ~/couchbase/install/bin $ ./cbindex \ -auth Administrator:asdasd -type list -server localhost:9000 List of indexes: Change-Id: I5d0242fc61db7695cdfe7c8dedcade9f6fb9b86b Reviewed-on: http://review.couchbase.org/114389 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Tested-by: Steve Watanabe <steve.watanabe@couchbase.com> Reviewed-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> * Commit: 0a96125775b6838c16081d83306288b34ece963a in build: 6.5.0-4228 move getStatsDirectory to mnStatisticsService Change-Id: Ia7ce36af68e6a93035ad67913b69641cb3c204b8 Reviewed-on: http://review.couchbase.org/114183 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: 18ac4c2e194b8dfc80a73d4178e3c3e8b464c1d0 in build: 6.5.0-4228 use statisticsDescriptionService for rendering stats in chart buidler Change-Id: I97c17bcbd479d578021563c17d712a2f564c4e4f Reviewed-on: http://review.couchbase.org/114018 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Pavel Blagodov <stochmail@gmail.com> Tested-by: Pavel Blagodov <stochmail@gmail.com> * Commit: 442c803a6b51c1343dfc8fc81b80f063d8816b26 in build: 6.5.0-4227 Remove unnecessary quotes. Change-Id: I9ed3c6c6cd69eabb51942c8cab494910f7871c2d Reviewed-on: http://review.couchbase.org/114409 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Steve Watanabe <steve.watanabe@couchbase.com> * Commit: c474f2a1e1b9e45d106c943ce31968f60bb2d6eb in build: 6.5.0-4226 eliminate some unnecessary foldl calls in node_monitor Change-Id: Idc1169e9ac4af2cc6489c04febb671aba835dc51 Reviewed-on: http://review.couchbase.org/114395 Tested-by: Artem Stemkovski <artem@couchbase.com> Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> * Commit: bc102ba18d9e7f8bdd35eda5dbac6c668e6135b2 in build: 6.5.0-4226 Merge remote-tracking branch 'couchbase/alice' into merge Empty merge, since both patches were independently submitted against master branch too. * couchbase/alice: MB-35686 - Remove hard-coded link to 5.0 upgrade instructions. MB-35840 - View status not appearing in UI. Change-Id: I885cb04719edba8780554a95ee22728acce478a1 * Commit: 18e1fc78fc05f7b1d19ac7c514357747961e21f4 in build: 6.5.0-4226 Merge commit 'couchbase/alice~~' into merge * commit 'couchbase/alice~~': MB-35748 Kill leader_lease_acquirer worker processes brutally. Remove an unneeded space. MB-35748 Flush {new_nodes, _} messages in leader_lease_acquirer. MB-35748 Prevent a deadlock in async termination. MB-35748 Factor out misc:terminate/2. MB-35748 Don't produce a crash on nack from register_child_async. MB-35748 Don't kill async on late registration request. MB-35748 Add a type spec to async_loop_with_result/1. MB-35748 Don't expect "future" requests. MB-35748 Drop unused monitor_pids async option. MB-35748 Drop unused async:adopt/1. MB-35748 Fix rebalance stop when bucket mover exits on its own. Change-Id: I346b4ebf59750db02e3092853aa68c22e32f2375 * Commit: 9ea01810bdc34ad923fe8431c508f26d448c9f2d in build: 6.5.0-4225 Merge commit 'a4d3a724ccc53e3e8f5d6257d7bebb7ef68fc4ed' into merge This is an empty merge since the fix for the same issue was introduced on the master branch independently. * commit 'a4d3a724ccc53e3e8f5d6257d7bebb7ef68fc4ed': MB-35686 - Remove hard-coded upgrade instructions link. Change-Id: I99fe5772b149b805ecf8dceeff9abc24ad6b70b1 * Commit: 1146d749ad4071aebd4c6ef3ba83e7fb7f01e86c in build: 6.5.0-4224 MB-35686 - Remove hard-coded link to 5.0 upgrade instructions. Instead follow the link to "current", which shoud be up to date. Change-Id: I01c59968f54a83fcbff45058e7c7a9b55fa4d7c2 Reviewed-on: http://review.couchbase.org/114351 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 9bb1d65bd8a71bcbeba2c9db5ad91b1eb44f8488 in build: 6.5.0-4226 MB-35686 - Remove hard-coded link to 5.0 upgrade instructions. Instead follow the link to "current", which shoud be up to date. Previous fix used downloads link, not upgrade instructions. Change-Id: I5b4d60f6af2923f9660fd796e1ac0a963bb09ce8 Reviewed-on: http://review.couchbase.org/114390 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 54c31e0657c2cbb6369702f8b7662431e5d0f960 in build: 6.5.0-4223 MB-35840 - View status not appearing in UI. Problem was that column for view status was set to flex-grow-0. Changing it to flex-grow-1 makes it visible. Change-Id: I1b50951f9cb45f4215e1f4d09f0767b552cf0c61 Reviewed-on: http://review.couchbase.org/114355 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: fe303d732d0ca00999fd712bf9181ba2327731d1 in build: 6.5.0-4226 MB-35840 - View status not appearing in UI. Problem was that column for view status was using class flex-grow-0. Changing it to flex-grow-1 makes it visible. Change-Id: I7e80fe97759a7896e03b5aa711d5e5f5dea7a832 Reviewed-on: http://review.couchbase.org/114388 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 7ae566cc2d83bd313416c7d3746667af50d99a7b in build: 6.5.0-4226 MB-35748 Kill leader_lease_acquirer worker processes brutally. It's suspected that due to a race condition in erlang runtime, leader_lease_acquirer failed to terminate one of its workers in CBSE-7275. Which lead to further unavailability of cluster operations requiring a majority of leases. [1] addresses the race condition. But it's also the case that there's no need for leader_lease_acquirer to terminate the workers gracefully. So this patch simply kills the workers brutally. [1] https://github.com/couchbasedeps/erlang/commit/c52409a88130cc435abf1cf81499c5b0e6bd1442 Change-Id: Id2e7f640fe022ae0bbbba488d7857d6d9b1b2abf Reviewed-on: http://review.couchbase.org/114295 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: f12511b94464d2d63a802ca68a44f476a11656c0 in build: 6.5.0-4226 Remove an unneeded space. Change-Id: Iaa4c532de71507097d54e975fa295cdb8d1c0a6f Reviewed-on: http://review.couchbase.org/114294 Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 3f2034c62e57c927b71193ef940752ff5f465f38 in build: 6.5.0-4226 MB-35748 Flush {new_nodes, _} messages in leader_lease_acquirer. Generally, we don't expect to see many {new_nodes, _} messages at the same time. But if network is flaky and the process is slow processing the messages, they might accumulate. So to stay on the safe side, let's make sure we flush all of the outstanding messages and process just the last one. Change-Id: I34ec580617b0fc1b041e5b18d69d8a66338f0eb9 Reviewed-on: http://review.couchbase.org/114293 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 8feff5c8db7001abcb16b3e9afe885813c986d07 in build: 6.5.0-4226 MB-35748 Prevent a deadlock in async termination. The following sequence of events leads to a deadlock. - An async (A) is spawned. The body function sets trap_exit to true. - async:abort(A) is called. A's controlling process sends an exit signal to the executor process and waits for it to terminate. - Since the executor process traps exits, it continues to run and spawns a new async B. - The controlling process of B attempts to register with A and blocks. - The body of A calls async:wait(B). Now A's controlling process waits for A's executor to terminate. A's executor waits for B. And B waits for A's controlling process. Similar deadlock may arise if the body of async A simply consumes an EXIT message without actually exiting. That's what happened in CBSE-6423 where ns_rebalancer was the process to consume the EXIT message (fixed by one of the previous changes). To address the issue, the controlling process continues to reply to registration requests even when it waits for children to terminate. Change-Id: I3c6c20d46ca7af1e7b6e226b97b2a880dbd420fd Reviewed-on: http://review.couchbase.org/114292 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: e853222230294f1bc64e94b3978ba27318ae3578 in build: 6.5.0-4226 MB-35748 Factor out misc:terminate/2. That acts like erlang:exit/2, but also terminates the process even if the reason passed is normal. Change-Id: I6a2d03687234adaf22b03f685ad4324f8425e5db Reviewed-on: http://review.couchbase.org/114291 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 1f1b6fc4ff0bd3cf5cce017c2a4c3db563bcac0e in build: 6.5.0-4226 MB-35748 Don't produce a crash on nack from register_child_async. Since it's a semi-expected situation. Change-Id: I1ac2a89cc64328906b4999d60dcabc68b7215196 Reviewed-on: http://review.couchbase.org/114290 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: d64d8b1a0a67584f4d26196c4bec4532f7175c80 in build: 6.5.0-4226 MB-35748 Don't kill async on late registration request. Change-Id: Ib1d04eeb306b1a09de9b81c48b452dfbc68b21cd Reviewed-on: http://review.couchbase.org/114289 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 776e8ee610d785afb65bd92bcc578e92147a2bce in build: 6.5.0-4226 MB-35748 Add a type spec to async_loop_with_result/1. Otherwise dialyzer starts complaining about the next change. In order for the return type to be no_return(), I also had to change handle_get_result/2 to do exit(normal) after replying to the caller. This was the effective behavior anyway, since async_loop_with_result/1 is only called in a tail position in the controlling process. Change-Id: Ib8e196facb1c945e87a537cb1c8f8138eb496c12 Reviewed-on: http://review.couchbase.org/114288 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: bda056d64eb1cfc82cfe69bea4d96ce019d90c97 in build: 6.5.0-4226 MB-35748 Don't expect "future" requests. Change-Id: Idf24c5e771897e7bb10090ac4446b6424823f26c Reviewed-on: http://review.couchbase.org/114287 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 2a5a1fbd0bf0bba95e60dd97f7bfd8dd6ec60380 in build: 6.5.0-4226 MB-35748 Drop unused monitor_pids async option. Change-Id: Id9d1ec4c02fbc0bde6142f2ba6f50257c47875d1 Reviewed-on: http://review.couchbase.org/114286 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: d1db8d4fc00b79cf93fd958d34f5b257a127e197 in build: 6.5.0-4226 MB-35748 Drop unused async:adopt/1. Change-Id: Ia0ebcf2d10bd0ddaaa57c9c7300e5c8e67a7e728 Reviewed-on: http://review.couchbase.org/114285 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: 007a59484799ed2d5a572ba274cf5a7980a7634f in build: 6.5.0-4226 MB-35748 Fix rebalance stop when bucket mover exits on its own. If rebalance stop request is sent just at the moment when one of the bucket movers terminates, the stop request may get ignored. The issue is in wait_for_mover function. When it receives a stop request from the parent, it sends an exit to the mover and simply calls itself recursively to wait till the mover exits. The problem is when the mover terminates independently with reason 'normal'. In this case, instead of exiting, wait_for_mover just returns 'ok', which continues rebalance. The solution is to disentangle waiting for the mover termination normally and when a stop request was received. Change-Id: I59bb0bcfadf8ae436f9bf371f292384b4653422a Reviewed-on: http://review.couchbase.org/114284 Well-Formed: Build Bot <build@couchbase.com> Tested-by: Aliaksey Artamonau <aliaksey.artamonau@couchbase.com> Reviewed-by: Artem Stemkovski <artem@couchbase.com> * Commit: a4d3a724ccc53e3e8f5d6257d7bebb7ef68fc4ed in build: 6.5.0-4225 MB-35686 - Remove hard-coded upgrade instructions link. Instead we should use the link that is helpfully provided by PhoneHome. Change-Id: I6f66455975f1a1a8580716394e62425d1867eb89 Reviewed-on: http://review.couchbase.org/113922 Well-Formed: Build Bot <build@couchbase.com> Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> CHANGELOG for plasma * Commit: f123aa5ec10a4bd3ae4a6ad35bea0cfb4069d028 in build: 6.5.0-4281 Merge remote-tracking branch 'couchbase-priv/unstable' into HEAD http://ci2i-unstable.northscale.in/gsi-13.09.2019-05.30.pass.html Change-Id: I35940def4701d49bab1f0905d3790b9f8051a614 * Commit: 307e69dbd8e7d9a8de92a7079174769a709b243d in build: 6.5.0-4281 MB-35907 Eliminate GetRecoveryPoints() unsafe behavior Change-Id: I0d8778d706ac38b47a7c4b1c0bb6c253948bc372 * Commit: cade44074e034f4c1df549c050444e8759432b66 in build: 6.5.0-4281 MB-35682 Reduce plasma logging for low memory situations Change-Id: I9f8419dc36b009ed6ca72a37a7e4c1287a1a5e4b * Commit: c36b955a14562f5e016a2a53d7f5cdacdab532a0 in build: 6.5.0-4246 Merge remote-tracking branch 'couchbase-priv/unstable' into HEAD http://ci2i-unstable.northscale.in/gsi-10.09.2019-14.16.pass.html Change-Id: I6c5eff492735903be61c1a1409a411212f06791d * Commit: 1c9f2aba4d7c8c9f010754995c6275331a55b364 in build: 6.5.0-4246 MB-34742: Persist memQuota in checkpoint memQuota is reset to user set quota after restart. To avoid this, persist memQuota in checkpoint. When we recover, set memQuota to the lowest quota found in the checkpoints and let tuner increment as needed. Change-Id: I843ce780233dc676babfbdc416b2a392c9e50a6a * Commit: 81b7f34dc880c55489efbc93e2dc106953bb88a0 in build: 6.5.0-4246 MB-35642: Evict dirty page when persist Evict dirty page at time of persit that is not used for scanning Evict a page immediately when calling Persist() if it is not used for scanning. Persist() can be called at persistent snapshot and swapper/eviction. There are 4 workload considered: 1) Scan, Persist, Mutate. The page is not evicted. This is existing behavior. 2) Mutate, Persist, Scan. The page is swapIn during scan. This is existing behavor, except that there is one more page segment to read from disk. 3) No correlation between mutate and scan. The mutated partial page will be evicted immediately upon Persist(). 4) Initial Index Build (mutation only). After Persist(), new page segment will need to be created for subsequent changes, so no need to keep the old data around. Note: 1) Besides Persist(), there are other places where a page can be flushed (e.g split/merge). Those pages will not be evicted. 2) Besides scan, there are other places where a page can be swapIn (e.g. split/merge). Those pages can be evicted upon mutated. 3) There are other places where a page will be read, but not swapIn (e.g. LookupKV). Those pages can be evicted upon mutated. Change-Id: I7262a94cf272d5cf4db67c343b606fcf9ba3805c * Commit: 5b4db1cbafec635e9fc217e57d9d9b42989f6e2d in build: 6.5.0-4246 MB-100: docs: Add plasma stats descriptions doc Change-Id: Id2f2d63c181faad970389fae0e1c12eac84ac07a * Commit: e5bf4f35b8d7d626e944209b370606f5c393f760 in build: 6.5.0-4246 MB-34084: Avoid fatal error on eviction timeout Indexer cleans up data files on fatal error during recovery. Avoid returning fatal error on eviction timeout due to low quota as this can be fixed by increasing the quota. Change-Id: Ibd3ccf3f7d17cf675da4676ed14598e7e95f6852 * Commit: 820b5562655707e4c13dd757774dea8415d39893 in build: 6.5.0-4246 MB-100: stats: Fix SwapInConflicts and ReclaimSzIndex - Increment SwapInConflicts whenever we retry. A previous refactor of ReadPage had removed this. - Add ReclaimSzIndex correctly during stats merge. Change-Id: Iba2a07326b8a57e35a6ee33b34b76742990750a8 * Commit: c09b02639bdaef1b18ac450ce7c48e690811d413 in build: 6.5.0-4246 MB-31787 Expose CheckpointSize in GetStats Change-Id: Id985e945bd3b482ae004045ded4a3cdbc8742749 * Commit: a4bef6e8606a5356c206bcdf68f2ed732bc665b9 in build: 6.5.0-4246 MB-35603: keep page level statistics up-to-date calculate page level statistics only for memory-resident pages calculate page level statistics only for full page (non-evicted page) increment page level statistics when compact, swapIn, recovery decrement page level statistics when free page page level statistics include delta records persist activeMaxPageItems, activeMinPageItems, activeDeltaChainLen in checkpoint Change-Id: I2846369923b5c4689645b8fc178ef4c7d15f2f7d CHANGELOG for query * Commit: db3aa471451fd4527fd1e3a4bcc2f45f102daf8e in build: 6.5.0-4240 MB-35873 Add testcases for array predicates Change-Id: I80063b49b2bb39d036a1025b137703ff3fcfc976 Reviewed-on: http://review.couchbase.org/114501 Reviewed-by: Bingjie Miao <bingjie.miao@couchbase.com> Tested-by: Chang Liu <chang.liu@couchbase.com> * Commit: cf080d9c2e413d0bf2583cdae404796a53da8d49 in build: 6.5.0-4238 MB-35871 Add cost/cardinality info for unnest scan Change-Id: I1d9c8bf71d6c176d9be7dee104662637d0ee2f09 Reviewed-on: http://review.couchbase.org/114478 Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> Tested-by: Bingjie Miao <bingjie.miao@couchbase.com> * Commit: a34f4d3f71ae526532662d5cd2ea1feb188672e4 in build: 6.5.0-4219 MB-35838 sarg keys not properly set for unnest scan using multiple keys Change-Id: I86153e42d105256c45dff3a73d217c683cc50e4f Reviewed-on: http://review.couchbase.org/114362 Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> Tested-by: Bingjie Miao <bingjie.miao@couchbase.com> CHANGELOG for query-ee * Commit: dbf12520a39e86ef5fb2007095bea0c345f306d6 in build: 6.5.0-4240 MB-35873 Borrow same logic to validate covering array index Change-Id: Iaea0182d5e6cc9f11671dcfae27a7c081fdd973b Reviewed-on: http://review.couchbase.org/114492 Reviewed-by: Bingjie Miao <bingjie.miao@couchbase.com> Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> Tested-by: Chang Liu <chang.liu@couchbase.com> * Commit: dc328ea3a3bfa950be0d8ba58bb73c42104b4d59 in build: 6.5.0-4238 MB-35871 Add cost/cardinality info for unnest scan Change-Id: Ia15535f83af930adb961db3c0d543ecbfaaee302 Reviewed-on: http://review.couchbase.org/114479 Reviewed-by: Sitaram Vemulapalli <sitaram.vemulapalli@couchbase.com> Tested-by: Bingjie Miao <bingjie.miao@couchbase.com> CHANGELOG for query-ui * Commit: 2290b957d87fbf5091af6e1b314da5c277fd79c1 in build: 6.5.0-4278 MB-35857 - Auto-shrink too aggressive Right now the query workbench has a variable indicating whether the user is focused on the query, or on the results, and expands the relative sizes of those parts of the UI accordingly. This change makes the value of that variable persistent when you go to other screens. So if you click on Documents, then return, your focus will stay the same. This change also changes the focus if the user scrolls either the query or JSON results. I.e., if the user scrolls the query, and it is not focused, it will become focused and enlarge, and vice versa for the JSON results. Change-Id: I87607e965fea1bbe73efa37a00f407b0df3ad4a3 Reviewed-on: http://review.couchbase.org/114729 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: df35966364b7f7b336f4dd9a34e1a71a83c91a5d in build: 6.5.0-4276 MB-35850 - Fix tabular layout of complex structures. The datatable class was not correctly rendering polymorphic objects - items that might be arrays in some documents and objects in others. This, combined with a height issue in Chrome, lead to poor layout for certain results. Change-Id: I5c1971444786bba177e9162263a5c3209c0cad0f Reviewed-on: http://review.couchbase.org/114722 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 71bf7e6478863fd472e52ecf6ba6b08749cf20dd in build: 6.5.0-4275 MB-35951 - Doc Editor using doc ID when hidden The problem only arose for buckets with no primary index. There was a logic issue where the code was only checking whether the document ID was specified, but not whether the mode was set to individual Doc IDs or Doc ID ranges. Change-Id: I5385cf715440362122d894849230c30098dd5045 Reviewed-on: http://review.couchbase.org/114704 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: af7df283022e37c7270e80094ce819091c3188c9 in build: 6.5.0-4265 MB-35850 - update table and tree views when multiple queries. The directives for table and tree views use the angular $watch to keep track of when the underlying data changes. Pushing new results onto the results array, however, wasn't recognized as a change, however, because the top-level array didn't change. With this change, we clone the result array, so that the directives notice the change. (The alternative was to tell the directive to do deep equality, but that could be very expensive with large results.) Change-Id: Ida3b068df167adfeefd2700bf58a98f62d7eebc4 Reviewed-on: http://review.couchbase.org/114634 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: e404de5960d374b395ac3198b5a35831b17758fd in build: 6.5.0-4254 revise query monitor footer markup for tiny charts Change-Id: Id12ba33377352cad6676362d0f4e448f0ca97fe8 Reviewed-on: http://review.couchbase.org/114576 Reviewed-by: Rob Ashcom <rob.ashcom@gmail.com> Tested-by: Rob Ashcom <rob.ashcom@gmail.com> * Commit: 9b6730a96b2d0062c44f8d5d2ca1b86d8a655c77 in build: 6.5.0-4252 WIP: Switch query monitoring charts to use new framework. Change-Id: Ieec6caac12f6d0a780a51991d16356a38bd999e6 Reviewed-on: http://review.couchbase.org/114179 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: e6a377f835a76c2fbebb0ab7d83e5c3f7c11d1a4 in build: 6.5.0-4252 For some reason the 'indexStatusURIChanged' event is firing all the time when queries are running. Don't update the UI when this happens. Change-Id: Ie0b65d76ff20f989df90c0cb4168779fd7f7f9ea Reviewed-on: http://review.couchbase.org/114567 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> * Commit: 9e898a387a1d62c3b3c6f3bc61463acbfe284b3f in build: 6.5.0-4252 MB-35798 - reduce the number of calls to the query service for metadata Prior to this change, the qwQueryService was making many REST calls to the query service to get the metadata for displaying in the query workbench, and for use in the document editor. For initially testing authorization on a bucket, it would call "select count(*) from <bucket>", and if that succeeded it would proceed to get the inferred schema. After that, every 5 seconds it would issue a "select count(*)" query for each bucket to keep the document counts up to date. With this change, the schema inference call is used to test authorization on a bucket, and a single N1QL query is used to retrieve the document count for every bucket. This reduces the number of initial REST calls to 2 + #buckets. $timeout is also used to slow down the rate of initial metadata queries. Change-Id: Ideab6828dbd68c381c4deb0ed34343d5cca33ebc Reviewed-on: http://review.couchbase.org/114566 Reviewed-by: Eben Haber <eben@couchbase.com> Tested-by: Eben Haber <eben@couchbase.com> CHANGELOG for sigar * Commit: eeb8b58411ebf8a5136570d3473bf255fed1f281 in build: 6.5.0-4220 MB-34865: let sigar_os_proc_list_get only return my processes sigar_port wants information about the child processes of the babysitter, but the API to sigar won't let it just query just that. Instead it needs to walk over all processes and look at the parent processes (but in order to fetch the ppid it collects a lot of other stuff). In sigar_os_proc_list_get we've got both the ppid and the uid available, but to avoid changing too much in the code I figured we could skip all process _not_ belonging to ourselves and then hopefully get rid of any access restrictions later on. Change-Id: I0d38ce5b91c471cac51e951a50e4062870c2ae30 Reviewed-on: http://review.couchbase.org/113948 Reviewed-by: Dave Rigby <daver@couchbase.com> Tested-by: Build Bot <build@couchbase.com> Reviewed-by: Daniel Owen <owend@couchbase.com> CHANGELOG for testrunner * Commit: e81b1288c60f41d52e41a8382db8db1d06e71eb5 in build: 6.5.0-4270 Revert "add debug log for subdoc error" This reverts commit d4820b423ba0e5e7937eef2c2635b594d2b3a953. Change-Id: Ie1ee5802e8ffd37e11359cd07e1e6a452c1ca7d8 Reviewed-on: http://review.couchbase.org/114662 Reviewed-by: Balakumaran G <balakumaran.gopal@couchbase.com> Tested-by: Balakumaran G <balakumaran.gopal@couchbase.com> * Commit: deef4cf5a2a9dc4ea0868716307e8076e2f1aaa0 in build: 6.5.0-4270 Return usage of primary index for specific jobs. Example - pushdown_primary Change-Id: I05d0a9468d28907a644d94f9f78e3d523a4b2758 Reviewed-on: http://review.couchbase.org/114654 Tested-by: <evgeny.makarenko@couchbase.com> Reviewed-by: Mihir Kamdar <mihir.kamdar@couchbase.com> * Commit: d4820b423ba0e5e7937eef2c2635b594d2b3a953 in build: 6.5.0-4269 add debug log for subdoc error Change-Id: I26529e72a75a606faa49c779fd668578e16f0fe3 Reviewed-on: http://review.couchbase.org/114645 Tested-by: Thuan Nguyen <soccon@gmail.com> Reviewed-by: Korrigan Clark <korrigan.clark@couchbase.com> * Commit: ae1cc9309bf6c9db4dc1a62ff117741828e196af in build: 6.5.0-4249 fix for window functions rqg jobs Change-Id: Id6d6cf67d08fa9b72f17bdd877d06822b1831a01 Reviewed-on: http://review.couchbase.org/114557 Reviewed-by: Mihir Kamdar <mihir.kamdar@couchbase.com> Tested-by: <evgeny.makarenko@couchbase.com> * Commit: 262dcea97ae326ba11db19e18e16d05c0bedb83f in build: 6.5.0-4240 fix for centos-rqg_ansi_transform_6.5_P1 and centos-rqg_fields_asc_desc_6.5_P1 Change-Id: I0668e19f4db3260b467b6d8c3acfad809a500469 Reviewed-on: http://review.couchbase.org/114500 Tested-by: <evgeny.makarenko@couchbase.com> Reviewed-by: Ajay Bhullar <ajay.bhullar@couchbase.com> * Commit: 9fb4edd45342683efade4a591abd9a37dafb17c6 in build: 6.5.0-4240 CRUD operations rqg jobs fix Change-Id: Ieac4cfa377bc07fb621f5cb93990e4c7a6acab2e Reviewed-on: http://review.couchbase.org/114392 Reviewed-by: Ajay Bhullar <ajay.bhullar@couchbase.com> Tested-by: <evgeny.makarenko@couchbase.com> * Commit: 89b0cc51bf86dd29e0dbe8bb36a3d5b0a5fc85fe in build: 6.5.0-4230 Adding seprate kv erlang test Change-Id: Ia63a23b38a13b94c869fa5c940c9cb655e6b1908 Reviewed-on: http://review.couchbase.org/114429 Reviewed-by: Balakumaran G <balakumaran.gopal@couchbase.com> Tested-by: vikas chaudhary <vikas.chaudhary@couchbase.com> * Commit: 92d8aad09761a22733c91016771cd1c0de634465 in build: 6.5.0-4227 Fixing dmg install command for mac Change-Id: I9431f8822c41d332e553cfe76b677c434844cd4c Reviewed-on: http://review.couchbase.org/114408 Tested-by: Arunkumar Senthilnathan <arun.couchbase@yahoo.com> Reviewed-by: <ofer.yaakobovitz@couchbase.com> CHANGELOG for tlm * Commit: e94ed5ce03fe73c8ce12b04cf8429d6475183f5a in build: 6.5.0-4282 Do not link to non-existent ICU libs Change-Id: I70e984f3946b3025f0e25f469a2490727be4919a Reviewed-on: http://review.couchbase.org/114739 Reviewed-by: Chris Hillery <ceej@couchbase.com> Tested-by: Gautham B A <gautham.bangalore@gmail.com> CHANGELOG for voltron * Commit: 975b48ebdad5dd159360f353af9125e939a0653a in build: 6.5.0-4253 MB-34977: Prevent zombie couchbase-server processes from zypper in container Something about how systemd-ctl forks off the couchbase-server script, which then exec's erl, combined with how zypper manually handles rpm %posttrans scripts rather than letting rpm do it itself, leads to the couchbase-server script process remaining as a zombie when installing via zypper inside a Docker container. This also results in zypper hanging indefinitely on install. This fix introduces a new subshell in the systemd-ctl script, which is necessary to correctly reap child processes. I don't fully understand it, but it works. I also fixed the stderr redirect from couchbase-server so no output can end up on the zypper console. Change-Id: Idb71ff9f5ace3d07fd8444b67e34b264b46f0881 Reviewed-on: http://review.couchbase.org/114517 Reviewed-by: Chris Hillery <ceej@couchbase.com> Tested-by: Chris Hillery <ceej@couchbase.com>