Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40301

update-eventing-function for Eventing CLI

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.5.1
    • 7.0.0
    • documentation, eventing
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      In the UI, We can pause-edit-resume a function.
      On the CLI, We could pause-import-resume a function.

       

      However, "import" seems to overwrite a function and lose its place in the source bucket. This would forces to "resume-from-now" and miss some data or go back to the vert beginning.
      There doesn't seem to be an "update-eventing-function" on the cli which matches the "edit Javascript" on the UI.

      another issue we are having is not knowing then the eventing function has completed its action. Since the commands seem to be asynchronous, we cannot script the commands

       

      undeploy func1

      delete func1

      import func1

       

      The import fails, since we don't know when the function is deleted. We need something like a --wait flag, which would make the execution synchronous.

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-4272 contains eventing commit 6fe8953 with commit message:
            MB-40301: Add /appcode endpoint to retrieve/update function code

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4272 contains eventing commit 6fe8953 with commit message: MB-40301 : Add /appcode endpoint to retrieve/update function code
            jon.strabala Jon Strabala added a comment -

            Vinayaka Kamath running a new 7.0 source build on March 12 2021, I can export just the appcode in a paused state, I can get the checksum, but I can not update the appcode I get a "Function compilation failed" error.  Below I use --data as an argument to the curl command

             

            # curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o ./appcode.orig
             
            # curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode/checksum' 
            fcb6a46866d8c143a1f93245cfe85468a66956866c85212835563bb0f4400949
             
            # curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @./appcode.orig
            {
             "name": "ERR_HANDLER_COMPILATION",
             "code": 27,
             "description": "Function compilation failed",
             "attributes": null,
             "runtime_info": {
             "code": 27,
             "info": "Function: timer_test appcode stored in the metakv."
             }
            }
            

            Howe er I can get update the function via the curl --data-binary flag instead of --data 

            # cat appcode.new
            curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --data-binary @./appcode.orig
            {
             "code": 0,
             "info": "Function: timer_test appcode stored in the metakv."
            }

            Since the appcode export is not binary why do I need to use this flag?

             

             

             

            jon.strabala Jon Strabala added a comment - Vinayaka Kamath  running a new 7.0 source build on March 12 2021, I can export just the appcode in a paused state, I can get the checksum, but I can not update the appcode I get a " Function compilation failed " error.  Below I use --data as an argument to the curl command   # curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o ./appcode.orig   # curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode/checksum' fcb6a46866d8c143a1f93245cfe85468a66956866c85212835563bb0f4400949   # curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @./appcode.orig { "name": "ERR_HANDLER_COMPILATION", "code": 27, "description": "Function compilation failed", "attributes": null, "runtime_info": { "code": 27, "info": "Function: timer_test appcode stored in the metakv." } } Howe er I can get update the function via the curl --data-binary flag instead of --data  # cat appcode.new curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --data-binary @./appcode.orig { "code": 0, "info": "Function: timer_test appcode stored in the metakv." } Since the appcode export is not binary why do I need to use this flag?      

            Jon Strabala This seems a bit strange. We don't need to pass --data-binary to upload the code, infact we never needed it. I tried reproducing your use case on my machine with a server running on cluster_run, works fine.

            ➜  eventing git:(resolve) curl -X GET -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode'
            function OnUpdate(doc, meta) {
                log("Doc created/updated", meta.id);
                let count = 10;
                var q1 = SELECT * FROM `dst` LIMIT $count;
                var q2 = SELECT * FROM `dst` LIMIT $count;
                var q3 = N1qlQuery("SELECT * FROM `dst` LIMIT 10");
                var q4 = "(SELECT * FROM `dst` LIMIT $count)";
            }
             
            function OnDelete(meta, options) {
                log("Doc deleted/expired", meta.id);
            }
            ➜  eventing git:(resolve) curl -X GET -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode' -o ./appcode.orig
            ➜  eventing git:(resolve) ✗ cat appcode.orig 
            function OnUpdate(doc, meta) {
                log("Doc created/updated", meta.id);
                let count = 10;
                var q1 = SELECT * FROM `dst` LIMIT $count;
                var q2 = SELECT * FROM `dst` LIMIT $count;
                var q3 = N1qlQuery("SELECT * FROM `dst` LIMIT 10");
                var q4 = "(SELECT * FROM `dst` LIMIT $count)";
            }
             
            function OnDelete(meta, options) {
                log("Doc deleted/expired", meta.id);
            }                                                                                                                                                                                                                                                           
            ➜  eventing git:(resolve) ✗ curl -X POST -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode' -d @./appcode.orig
            {
             "code": 0,
             "info": "Function: N1qlTest appcode stored in the metakv."
            }%                                                                               
            

            If I were to guess, could be that you're writing an escaped string into your file for some reason or your handler had some binary data. Can you please look into it in further detail Jon Strabala? What was your handler code consisting off?

            vinayaka.kamath Vinayaka Kamath (Inactive) added a comment - Jon Strabala This seems a bit strange. We don't need to pass --data-binary to upload the code, infact we never needed it. I tried reproducing your use case on my machine with a server running on cluster_run, works fine. ➜ eventing git:(resolve) curl -X GET -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode' function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); let count = 10; var q1 = SELECT * FROM `dst` LIMIT $count; var q2 = SELECT * FROM `dst` LIMIT $count; var q3 = N1qlQuery("SELECT * FROM `dst` LIMIT 10"); var q4 = "(SELECT * FROM `dst` LIMIT $count)"; }   function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); } ➜ eventing git:(resolve) curl -X GET -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode' -o ./appcode.orig ➜ eventing git:(resolve) ✗ cat appcode.orig function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); let count = 10; var q1 = SELECT * FROM `dst` LIMIT $count; var q2 = SELECT * FROM `dst` LIMIT $count; var q3 = N1qlQuery("SELECT * FROM `dst` LIMIT 10"); var q4 = "(SELECT * FROM `dst` LIMIT $count)"; }   function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); } ➜ eventing git:(resolve) ✗ curl -X POST -s 'http://Administrator:asdasd@localhost:9300/api/v1/functions/N1qlTest/appcode' -d @./appcode.orig { "code": 0, "info": "Function: N1qlTest appcode stored in the metakv." }% If I were to guess, could be that you're writing an escaped string into your file for some reason or your handler had some binary data. Can you please look into it in further detail Jon Strabala ? What was your handler code consisting off?
            jon.strabala Jon Strabala added a comment - - edited

            Vinayaka Kamathplease try with my Function (maybe it has binary chars) but it imports and deploys just fine.  So I am perplexed on why the appcode has an issue.

            Okay I uploaded "timer_test.json", make a bucket "source" and a bucker "meta" import it via the UI into Eventing

            linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out
            linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @/tmp/appcode.out
            {
             "name": "ERR_HANDLER_COMPILATION",
             "code": 27,
             "description": "Function compilation failed",
             "attributes": null,
             "runtime_info": {
             "code": 27,
             "info": "Function: timer_test appcode stored in the metakv."
             }
            }linuxbrew@couch01:~${noformat}
             
            

            But it will deploy and undeploy

            Now export and import the function it works

             

            linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test' -o /tmp/timer_test.json.exp
             linuxbrew@couch01:~$ vi !$
             vi /tmp/timer_test.json.exp
             linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test' -d @/tmp/timer_test.json.exp
             {
             "code": 0,
             "info":
            { "status": "Stored function: 'timer_test' in metakv", "warnings": null }
            }linuxbrew@couch01:~$
            

            Now try it again ...

             

            linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out
             linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @/tmp/appcode.out
             {
             "name": "ERR_HANDLER_COMPILATION",
             "code": 27,
             "description": "Function compilation failed",
             "attributes": null,
             "runtime_info":
            { "code": 27, "info": "Function: timer_test appcode stored in the metakv." }
            }linuxbrew@couch01:~$
            

             Then try --data-binary and it works for just the appcode

             linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --data-binary @/tmp/appcode.out{ "code": 0, "info": "Function: timer_test appcode stored in the metakv." }linuxbrew@couch01:~${noformat}

            The dd thing is the  "timer_test.json" file looks clean (no binary)

            linuxbrew@couch01:/tmp$ grep -Plv '[\0-\x7f]' /tmp/timer_test.json.exp
            linuxbrew@couch01:/tmp$ grep -Plv '[\0-\x7f]' /tmp/timer_test.json.exp | wc -c
            0

             

            jon.strabala Jon Strabala added a comment - - edited Vinayaka Kamath please try with my Function (maybe it has binary chars) but it imports and deploys just fine.  So I am perplexed on why the appcode has an issue. Okay I uploaded "timer_test.json", make a bucket "source" and a bucker "meta" import it via the UI into Eventing linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @/tmp/appcode.out { "name": "ERR_HANDLER_COMPILATION", "code": 27, "description": "Function compilation failed", "attributes": null, "runtime_info": { "code": 27, "info": "Function: timer_test appcode stored in the metakv." } }linuxbrew@couch01:~${noformat}   But it will deploy and undeploy Now export and import the function it works   linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test' -o /tmp/timer_test.json.exp linuxbrew@couch01:~$ vi !$ vi /tmp/timer_test.json.exp linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test' -d @/tmp/timer_test.json.exp { "code": 0, "info": { "status": "Stored function: 'timer_test' in metakv", "warnings": null } }linuxbrew@couch01:~$ Now try it again ...   linuxbrew@couch01:~$ curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -d @/tmp/appcode.out { "name": "ERR_HANDLER_COMPILATION", "code": 27, "description": "Function compilation failed", "attributes": null, "runtime_info": { "code": 27, "info": "Function: timer_test appcode stored in the metakv." } }linuxbrew@couch01:~$  Then try --data-binary and it works for just the appcode  linuxbrew@couch01:~$ curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --data-binary @/tmp/appcode.out{ "code": 0, "info": "Function: timer_test appcode stored in the metakv." }linuxbrew@couch01:~${noformat} The dd thing is the  "timer_test.json" file looks clean (no binary) linuxbrew@couch01:/tmp$ grep -Plv '[\0-\x7f]' /tmp/timer_test.json.exp linuxbrew@couch01:/tmp$ grep -Plv '[\0-\x7f]' /tmp/timer_test.json.exp | wc -c 0  

            Thanks Jon Strabala for helping me out with this. I was able to reproduce the scenario with your handler. Turns out that our parser handles escaped and unescaped strings differently and our appcode endpoint deals with unescaped strings in their raw form.

            The function code from the /appcode endpoint wasn't escaped during GET explicitly to improve readability (i.e we don't JSON Marshal it during the HTTP response). The idea is that the user can actually store and handle the appcode in files and edit it using any text editor.

            Coming to the issue that we're seeing, turns when you use a combination of --data and @ to read from a file, the carriage returns and newlines are stripped off from the data before sending it in the request. This won't be a problem when the data is JSON since the strings are escaped and stripping \n & \r only means messing with the indentation of the file. However this can cause problems with code since some lines might get commented out or it can alter the meaning of the code (this might depend on the code, the handlers which I was using didn't have any comments on them – this is why I was not able to reproduce the issue).

            On the contrary, using --data-binary doesn't strip \n and \r from the payload and the meaning remains intact. This is why using --data-binary works flawlessly. Refer curl documentation from here for more details

            Jon Strabala I think we should document the end point to use --data-binary instead of --data to avoid confusion while reading from a file here onwards. Be mindful that this is not the case when I am not using @ to read from a file. Using --data should work just fine if I'm passing a string.

            vinayaka.kamath Vinayaka Kamath (Inactive) added a comment - - edited Thanks Jon Strabala for helping me out with this. I was able to reproduce the scenario with your handler. Turns out that our parser handles escaped and unescaped strings differently and our appcode endpoint deals with unescaped strings in their raw form. The function code from the /appcode endpoint wasn't escaped during GET explicitly to improve readability (i.e we don't JSON Marshal it during the HTTP response). The idea is that the user can actually store and handle the appcode in files and edit it using any text editor. Coming to the issue that we're seeing, turns when you use a combination of --data and @ to read from a file, the carriage returns and newlines are stripped off from the data before sending it in the request. This won't be a problem when the data is JSON since the strings are escaped and stripping \n & \r only means messing with the indentation of the file. However this can cause problems with code since some lines might get commented out or it can alter the meaning of the code (this might depend on the code, the handlers which I was using didn't have any comments on them – this is why I was not able to reproduce the issue). On the contrary, using --data-binary doesn't strip \n and \r from the payload and the meaning remains intact. This is why using --data-binary works flawlessly. Refer curl documentation from here for more details Jon Strabala I think we should document the end point to use --data-binary instead of --data to avoid confusion while reading from a file here onwards. Be mindful that this is not the case when I am not using @ to read from a file. Using --data should work just fine if I'm passing a string.
            jon.strabala Jon Strabala added a comment - - edited

            Thanks Vinayaka Kamath so far --data-binary is only needed for api/v1/functions/timer_test/appcode right now and as you say this behaviour is important to document for 7.0.X and any potential backport to 6.6.X for CI/CD pipelines.

            It also appears that we can use --upload-file (or  the short argument -T ) to accomplish the same thing. I lean towards documenting this specific endpoint with --upload-file it is more intuitive (because in my mind the file is NOT binary) - 

            root@couch01:~# curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out
             
            root@couch01:~# curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --upload-file /tmp/appcode.out
            {
             "code": 0,
             "info": "Function: timer_test appcode stored in the metakv."
            }
             
            root@couch01:~# curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out2
             
            root@couch01:~# diff /tmp/appcode.out /tmp/appcode.out2
            root@couch01:~#
            

             and we can load stuff from stdin too

            root@couch01:~# curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --upload-file - < /tmp/appcode.out
            {
             "code": 0,
             "info": "Function: timer_test appcode stored in the metakv."
            }

             

            jon.strabala Jon Strabala added a comment - - edited Thanks  Vinayaka Kamath  so far --data-binary is only needed for api/v1/functions/timer_test/appcode right now and as you say this behaviour is important to document for 7.0.X and any potential backport to 6.6.X for CI/CD pipelines. It also appears that we can use --upload-file (or  the short argument -T ) to accomplish the same thing. I lean towards documenting this specific endpoint with --upload-file it is more intuitive (because in my mind the file is NOT binary) -  root@couch01:~# curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out   root@couch01:~# curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --upload-file /tmp/appcode.out { "code": 0, "info": "Function: timer_test appcode stored in the metakv." }   root@couch01:~# curl -X GET -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' -o /tmp/appcode.out2   root@couch01:~# diff /tmp/appcode.out /tmp/appcode.out2 root@couch01:~#  and we can load stuff from stdin too root@couch01:~# curl -X POST -s 'http://'${CB_USERNAME}:${CB_PASSWORD}'@localhost:8096/api/v1/functions/timer_test/appcode' --upload-file - < /tmp/appcode.out { "code": 0, "info": "Function: timer_test appcode stored in the metakv." }  
            sujay.gad Sujay Gad added a comment -

            Verified using 7.0.0-4854
            /appcode endpoint can be used to update function code.

            sujay.gad Sujay Gad added a comment - Verified using 7.0.0-4854 /appcode endpoint can be used to update function code.

            People

              vinayaka.kamath Vinayaka Kamath (Inactive)
              karthik.sekar Karthik B Sekar
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty