Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
Cheshire-Cat
-
1
-
Yes
-
KV-Engine 2021-March
Description
Eventing seems broken in new builds since Apr 12 18:23 PST via source builds.
Simple functions do not process mutations after deploying, and the metadata is not cleaned up on undeploy.
Example deploy a basic function (copy items from 'source' to 'destination' and enrich.
function OnUpdate(doc, meta) {
|
// 1. get integer timestamp in millis
|
var ts = Date.now();
|
// 2. add or overwrite the property
|
doc.cur_ts = ts;
|
// 3. update peer document in a different bucket
|
log(ts,doc,meta.id);
|
dst_bkt[meta.id] = doc;
|
}
|
Make a mutation by creating a document in the 'source' bucket the function will not peg 'success"
Undeploy the function the metadata Eventing scratchpad is not cleared leaving 1024 docs.
The current build Couchbase Server 7.0.0-4942 (EE) also fails. I took a cbcollect_info
curl -v --upload-file /space/cbcollect_info_20210414_eventing_7.0.0-4942_EE_broke.zip https://s3.amazonaws.com/customers.couchbase.com/jonstrabala/
|
In eventing.log I seem to get an error sequence for for every vBucket
Builds for the past three days have lots of [Error] messages as in 9000+ normally we have three (3) such messages
grep '\[Error' eventing.log | wc -l
|
9803
|
grep '\[Error' eventing.log | egrep '(STREAMREQ.*EINVAL)' | wc -l
|
9800
|
the first three [Error] messages in eventing.log below are expected but the other 9800 are not.
grep '\[Error' eventing.log | head -4
|
2021-04-14T07:36:55.884-07:00 [Error] Client::Get URL: http://127.0.0.1:8096/getAggRebalanceProgress Failed to set auth params, err: Unable to find given hostport in cbauth database: `127.0.0.1:8096'
|
2021-04-14T07:36:55.884-07:00 [Error] util::GetProgress Failed to gather task status from url: http://127.0.0.1:8096/getAggRebalanceProgress, err: Unable to find given hostport in cbauth database: `127.0.0.1:8096'
|
2021-04-14T07:37:55.177-07:00 [Error] util::ReadAppContent Function: test_issue app content doesn't exist or is empty
|
2021-04-14T07:38:16.234-07:00 [Error] DCPT[eventing:RqVgWciJ-4:{eventing:RqVgWciJ-3:worker_test_issue_0_127.0.0.1:11210_127.0.0.1:8096}/0] ##0 STREAMREQ(0) unexpected status: EINVAL
|
grep '\[Error' eventing.log | tail -1
|
2021-04-14T07:52:13.162-07:00 [Error] DCPT[eventing:RqVgWciJ-4:{eventing:RqVgWciJ-3:worker_test_issue_0_127.0.0.1:11210_127.0.0.1:8096}/0] ##245 STREAMREQ(581) unexpected status: EINVAL
|
A full unfiltered logging sequence this seems to repeat for each vBucket
2021-04-14T07:38:17.294-07:00 [Info] Consumer::processDCPEvents [worker_test_issue_0:/tmp/127.0.0.1:8091_0_1065054607.sock:8024] vb: 169 STREAMREQ Failed. Inserting entry: &consumer.vbFlogEntry{flog:(*memcached.FailoverLog)(nil), seqNo:0x0, signalStreamEnd:false, statusCode:0x4, streamReqRetry:true, vb:0xa9} to vbFlogChan
|
2021-04-14T07:38:17.294-07:00 [Info] Consumer::processDCPEvents [worker_test_issue_0:/tmp/127.0.0.1:8091_0_1065054607.sock:8024] vb: 170 got STREAMREQ status: EINVAL
|
2021-04-14T07:38:17.294-07:00 [Info] Consumer::processDCPEvents [worker_test_issue_0:/tmp/127.0.0.1:8091_0_1065054607.sock:8024] vb: 170 STREAMREQ metadataUpdated: true
|
2021-04-14T07:38:17.294-07:00 [Error] DCPT[eventing:RqVgWciJ-4:{eventing:RqVgWciJ-3:worker_test_issue_0_127.0.0.1:11210_127.0.0.1:8096}/0] ##10f STREAMREQ(271) unexpected status: EINVAL
|
2021-04-14T07:38:17.294-07:00 [Info] Consumer::processDCPEvents [worker_test_issue_0:/tmp/127.0.0.1:8091_0_1065054607.sock:8024] vb: 170 STREAMREQ Failed. Inserting entry: &consumer.vbFlogEntry{flog:(*memcached.FailoverLog)(nil), seqNo:0x0, signalStreamEnd:false, statusCode:0x4, streamReqRetry:true, vb:0xaa} to vbFlogChan
|
I uploaded a test case of two files just initialize a server wih eventing and at least 300 MB od Data Quota (no buckets) then setup up the EVARs in script ./test_issue.sh and finally run it.
test_issue.sh and test_issue.json
Abnormal failure (from Couchbase Server 7.0.0-4942) running ./test_issue.sh
we will get a LCB_ERR_DOCUMENT_NOT_FOUND because Eventing did not work.
root@couch01:/space/linuxbrew/eventing_broke# ./test_issue.sh
|
========================================================
|
MAKE 3 BUCKETS: source, meta, destination
|
SUCCESS: Bucket created
|
SUCCESS: Bucket created
|
SUCCESS: Bucket created
|
========================================================
|
IMPORT EVENTION FUNCTION : test_issue from test_issue.json
|
SUCCESS: Events imported
|
========================================================
|
ADD ONE DOC mykey to BUCKET source
|
mykey Stored. CAS=0x1675cbf206f90000
|
SYNCTOKEN=102,130335951495995,1
|
========================================================
|
LOOK FOR DOC mykey in BUCKET source
|
mykey CAS=0x1675cbf206f90000, Flags=0x0, Size=11, Datatype=0x01(JSON)
|
{"mykey":1}
|
========================================================
|
DEPLOY test_issue
|
========================================================
|
sleep 20 sec.
|
========================================================
|
LOOK FOR DOC mykey in BUCKET destination - we shold get the same as source but with a new property somehting like 'cur_ts':1618422868663
|
mykey LCB_ERR_DOCUMENT_NOT_FOUND (301)
|
Normal expected output (from Couchbase Server 7.0.0-4919) running ./test_issue.sh
root@couch01:/space/linuxbrew/eventing_broke# ./test_issue.sh
|
========================================================
|
MAKE 3 BUCKETS: source, meta, destination
|
SUCCESS: Bucket created
|
SUCCESS: Bucket created
|
SUCCESS: Bucket created
|
========================================================
|
IMPORT EVENTION FUNCTION : test_issue from test_issue.json
|
SUCCESS: Events imported
|
========================================================
|
ADD ONE DOC mykey to BUCKET source
|
mykey Stored. CAS=0x1675cbbd26c50000
|
SYNCTOKEN=102,188705952055070,1
|
========================================================
|
LOOK FOR DOC mykey in BUCKET source
|
mykey CAS=0x1675cbbd26c50000, Flags=0x0, Size=11, Datatype=0x01(JSON)
|
{"mykey":1}
|
========================================================
|
DEPLOY test_issue
|
========================================================
|
sleep 20 sec.
|
========================================================
|
LOOK FOR DOC mykey in BUCKET destination - we shold get the same as source but with a new property somehting like 'cur_ts':1618422868663
|
mykey CAS=0x1675cbc02f450000, Flags=0x0, Size=34, Datatype=0x01(JSON)
|
{"mykey":1,"cur_ts":1618423667395}
|
Other:
Note my last source build that worked was on April 7 07:46 PDT, and a more recent nas-server builds that worked
workes fine April 9 2021 Couchbase Server 7.0.0-4907 (EE)
i.e. couchbase-server-enterprise_7.0.0-4907-debian10_amd64.deb
workes fine April 11 2021 Couchbase Server 7.0.0-4919 (EE)
i.e. couchbase-server-enterprise_7.0.0-4919-debian10_amd64.deb
