Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.0.4
-
Untriaged
-
1
-
Yes
Description
I made an Eventing function "ttl2" in which some mutations are not processed despite a deploy from "Everything" oddly if I export it and then import it as "ttl3" it works as expected.
I have documents a, b, c, and d however as can be seen below for function "ttl2" document with KEY "a" doesn't trigger the Eventing handler. I only show the 6 lines of log file output and the undeployment from the prior run.
root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing# tail -7 ttl2.log
|
2022-10-04T13:05:45.231-07:00 [INFO] Function: ttl2 undeployed, reason: Function: ttl2 enabled, settings change requesting undeployment
|
2022-10-04T13:07:58.910-07:00 [INFO] "D 0-entry" {"cas":"1664912513582563328","id":"d","expiration":1954564032,"flags":0,"vb":221,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"d":1,"rand":0.2023158368195379}
|
2022-10-04T13:07:58.911-07:00 [INFO] "D 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:07:59.165-07:00 [INFO] "B 0-entry" {"cas":"1664912513859584000","id":"b","expiration":1954564032,"flags":0,"vb":446,"seq":963,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"b":1}
|
2022-10-04T13:07:59.165-07:00 [INFO] "B 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:07:59.472-07:00 [INFO] "C 0-entry" {"cas":"1664912514159149056","id":"c","expiration":1954564032,"flags":0,"vb":697,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"c":1}
|
2022-10-04T13:07:59.472-07:00 [INFO] "C 1 noop expiration is " 1954564032 "do nothing"
|
root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing#
|
Now if I export "ttl2" (via the UI) and import it as "ttl3" (via the UI) then deploy the function "ttl3" I see document with KEY "a" is processed now as expected.
root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing# tail -10 ttl3.log
|
2022-10-04T13:01:11.734-07:00 [INFO] "A 0-entry" {"cas":"1664912123250343936","id":"a","expiration":1954564032,"flags":33554438,"vb":183,"seq":94,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"a":1}
|
2022-10-04T13:01:11.735-07:00 [INFO] "A 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:01:11.759-07:00 [INFO] "D 0-entry" {"cas":"1664912513582563328","id":"d","expiration":1954564032,"flags":0,"vb":221,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"d":1,"rand":0.2023158368195379}
|
2022-10-04T13:01:11.759-07:00 [INFO] "D 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:01:12.025-07:00 [INFO] "B 0-entry" {"cas":"1664912513859584000","id":"b","expiration":1954564032,"flags":0,"vb":446,"seq":963,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"b":1}
|
2022-10-04T13:01:12.025-07:00 [INFO] "B 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:01:12.318-07:00 [INFO] "C 0-entry" {"cas":"1664912514159149056","id":"c","expiration":1954564032,"flags":0,"vb":697,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"c":1}
|
2022-10-04T13:01:12.319-07:00 [INFO] "C 1 noop expiration is " 1954564032 "do nothing"
|
2022-10-04T13:02:00.167-07:00 [INFO] Function: ttl3 undeployed, reason: Function: ttl3 enabled, settings change requesting undeployment
|
root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing#
|
I will attach both ttl2 and ttl2 (exported as follows)
/opt/couchbase/bin/couchbase-cli eventing-function-setup -c 192.168.3.150 -u admin -p aaa8938 --export --name ttl2 --file /tmp/ttl2.json
|
SUCCESS: Function exported to: /tmp/ttl2.json
|
/opt/couchbase/bin/couchbase-cli eventing-function-setup -c 192.168.3.150 -u admin -p aaa8938 --export --name ttl3 --file /tmp/ttl3.json
|
SUCCESS: Function exported to: /tmp/ttl3.json
|
And I will attach a cbcollect_info taken via with "ttl2" deployed after the condition where document a was not processed.
sudo /opt/couchbase/bin/cbcollect_info --tmp-dir=/tmp /tmp/odd_lost_mutation.zip
|
In addition I will provide a video of this anomaly. The first cbcollect was 170mb (too large) so I could not attach it to this MB
curl -v --upload-file odd_lost_mutation.zip https://s3.amazonaws.com/customers.couchbase.com/jonstrabala
|
As per the second Video it seems related to when an Evening Function creates documents in some cases DCP no longer find them??
The third video is from a source build on Oct 4, 2022 with a minimal system showing the incorrect behaviour. Along with an upload of both the Eventign function ttl2.json and the cbcollect_info: odd_lost_mutation2.zip from this source build.
Update - tested various versions and this seems to be a regression come 7.0.4
- Has the issue: Enterprise Edition 7.1.2 build 3437
- Has the issue: Enterprise Edition 7.1.2 build 3444
- Has the issue: Enterprise Edition 7.1.1 build 3175
- Has the issue: Enterprise Edition 7.1.0 build 2556
- Has the issue: Enterprise Edition 7.0.4 build 7279
- Works Fine: Enterprise Edition 7.0.3 build 7031
- Works Fine: Enterprise Edition 7.0.0 build 5302