Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53968

Unexplained lost mutations in Eventing

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      I made an Eventing function "ttl2" in which some mutations are not processed despite a deploy from "Everything" oddly if I export it and then import it as "ttl3" it works as expected.  

      I have documents a, b, c, and d however as can be seen below for function "ttl2" document with KEY "a" doesn't trigger the Eventing handler.   I only show the 6 lines of log file output and the undeployment from the prior run.

      root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing# tail -7 ttl2.log
      2022-10-04T13:05:45.231-07:00 [INFO] Function: ttl2 undeployed, reason: Function: ttl2 enabled, settings change requesting undeployment
      2022-10-04T13:07:58.910-07:00 [INFO] "D 0-entry" {"cas":"1664912513582563328","id":"d","expiration":1954564032,"flags":0,"vb":221,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"d":1,"rand":0.2023158368195379}
      2022-10-04T13:07:58.911-07:00 [INFO] "D 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:07:59.165-07:00 [INFO] "B 0-entry" {"cas":"1664912513859584000","id":"b","expiration":1954564032,"flags":0,"vb":446,"seq":963,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"b":1}
      2022-10-04T13:07:59.165-07:00 [INFO] "B 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:07:59.472-07:00 [INFO] "C 0-entry" {"cas":"1664912514159149056","id":"c","expiration":1954564032,"flags":0,"vb":697,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"c":1}
      2022-10-04T13:07:59.472-07:00 [INFO] "C 1 noop expiration is " 1954564032 "do nothing"
      root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing#

      Now if I export "ttl2" (via the UI) and import it as "ttl3" (via the UI) then deploy the function "ttl3" I see document with KEY "a" is processed now as expected.

      root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing# tail -10 ttl3.log
      2022-10-04T13:01:11.734-07:00 [INFO] "A 0-entry" {"cas":"1664912123250343936","id":"a","expiration":1954564032,"flags":33554438,"vb":183,"seq":94,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"a":1}
      2022-10-04T13:01:11.735-07:00 [INFO] "A 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:01:11.759-07:00 [INFO] "D 0-entry" {"cas":"1664912513582563328","id":"d","expiration":1954564032,"flags":0,"vb":221,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"d":1,"rand":0.2023158368195379}
      2022-10-04T13:01:11.759-07:00 [INFO] "D 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:01:12.025-07:00 [INFO] "B 0-entry" {"cas":"1664912513859584000","id":"b","expiration":1954564032,"flags":0,"vb":446,"seq":963,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"b":1}
      2022-10-04T13:01:12.025-07:00 [INFO] "B 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:01:12.318-07:00 [INFO] "C 0-entry" {"cas":"1664912514159149056","id":"c","expiration":1954564032,"flags":0,"vb":697,"seq":11,"datatype":"json","keyspace":{"bucket_name":"data","scope_name":"_default","collection_name":"_default"},"cid":0,"expiry_date":"2031-12-09T06:27:12.000Z"} {"c":1}
      2022-10-04T13:01:12.319-07:00 [INFO] "C 1 noop expiration is " 1954564032 "do nothing"
      2022-10-04T13:02:00.167-07:00 [INFO] Function: ttl3 undeployed, reason: Function: ttl3 enabled, settings change requesting undeployment
      root@couch01:/opt/couchbase/var/lib/couchbase/data/@eventing#

      I will attach both ttl2 and ttl2 (exported as follows)

      /opt/couchbase/bin/couchbase-cli eventing-function-setup -c 192.168.3.150 -u admin -p aaa8938 --export --name ttl2 --file /tmp/ttl2.json
      SUCCESS: Function exported to: /tmp/ttl2.json
      /opt/couchbase/bin/couchbase-cli eventing-function-setup -c 192.168.3.150 -u admin -p aaa8938 --export --name ttl3 --file /tmp/ttl3.json
      SUCCESS: Function exported to: /tmp/ttl3.json

      And I will attach a cbcollect_info taken via with "ttl2" deployed after the condition where document a was not processed.

      sudo /opt/couchbase/bin/cbcollect_info --tmp-dir=/tmp /tmp/odd_lost_mutation.zip

      In addition I will provide a video of this anomaly.  The first cbcollect was 170mb (too large) so I could not attach it to this MB

      curl -v --upload-file odd_lost_mutation.zip https://s3.amazonaws.com/customers.couchbase.com/jonstrabala

      As per the second Video it seems related to when an Evening Function creates documents in some cases DCP no longer find them??

      The third video is from a source build on Oct 4, 2022 with a minimal system showing the incorrect behaviour.  Along with an upload of both the Eventign function ttl2.json and the cbcollect_info: odd_lost_mutation2.zip from this source build.

      Update - tested various versions and this seems to be a regression come 7.0.4

      • Has the issue: Enterprise Edition 7.1.2 build 3437
      • Has the issue: Enterprise Edition 7.1.2 build 3444
      • Has the issue: Enterprise Edition 7.1.1 build 3175
      • Has the issue: Enterprise Edition 7.1.0 build 2556
      • Has the issue: Enterprise Edition 7.0.4 build 7279
      • Works Fine:   Enterprise Edition 7.0.3 build 7031
      • Works Fine:   Enterprise Edition 7.0.0 build 5302

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.2.0-2228 contains eventing commit d5e5658 with commit message:
            MB-53968 : Assign function instance id when going from undeploy ->

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-2228 contains eventing commit d5e5658 with commit message: MB-53968 : Assign function instance id when going from undeploy ->

            Build couchbase-server-7.2.0-5000 contains eventing commit 8a3712b with commit message:
            MB-53968 : Assign function instance id when going from undeploy ->

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5000 contains eventing commit 8a3712b with commit message: MB-53968 : Assign function instance id when going from undeploy ->

            Build couchbase-server-8.0.0-1132 contains eventing commit d5e5658 with commit message:
            MB-53968 : Assign function instance id when going from undeploy ->

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1132 contains eventing commit d5e5658 with commit message: MB-53968 : Assign function instance id when going from undeploy ->

            Sujay Gad Can you close this ticket by 10/10?

            chanabasappa.ghali Chanabasappa Ghali added a comment - Sujay Gad Can you close this ticket by 10/10?
            sujay.gad Sujay Gad added a comment -

            Verified the fix on 7.1.2-3450.
            Also added a functional test covering this scenario.

            sujay.gad Sujay Gad added a comment - Verified the fix on 7.1.2-3450. Also added a functional test covering this scenario.

            People

              sujay.gad Sujay Gad
              jon.strabala Jon Strabala
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty