Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39878

[BP MB-39877] - eventing crashes due to a race between undeploy and delete

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.5.1
    • Fix Version/s: 6.6.0
    • Component/s: eventing
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Ankit Prabhu diagnosed the issue on Slack

      So its a race condition between undeployment and delete. When we click on delete in ui it will 1st delete the tempstore and then proceed to delete from primary store.

      In primary store it will delete the settings path first
      and then check whether the app is undergoing undeployment or not
      https://github.com/couchbase/eventing/blob/5d15449549fcc7770fa1facdbddbbf063c0b5fb4/service_manager/http_handlers.go#L135

      service_manager/http_handlers.go:135
          err := util.MetaKvDelete(settingPath, nil)
      

      https://github.com/couchbase/eventing/blob/5d15449549fcc7770fa1facdbddbbf063c0b5fb4/service_manager/http_handlers.go#L146

      service_manager/http_handlers.go:146
              info.Info = fmt.Sprintf("Function: %s is currently undergoing undeploy. Only undeployed function can be deleted", appName)
      

      Now since its undergoing undeployment it will simply return and it won't delete the primary store.
      Now those checksome and definition remains in primary store
      When we rebalance other nodes then it will try to read the settings from setting path and it won't be able to read and the interface will be nil

      2020-06-07T09:35:28.007-05:00 [Error] ServiceMgr::parseFunctionPayload failed to unmarshal settings data from metakv, err: unexpected end of JSON input
      

      It will exit it by giving the interface conversion error.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            ritam.sharma Ritam Sharma added a comment -

            Jeelan Poola and Till Westmann - We can take this for 6.6

            Show
            ritam.sharma Ritam Sharma added a comment - Jeelan Poola and Till Westmann - We can take this for 6.6
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7811 contains eventing commit 99d5950 with commit message:
            MB-39878: Allow delete app if its being undeployed

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7811 contains eventing commit 99d5950 with commit message: MB-39878 : Allow delete app if its being undeployed
            Hide
            vikas.chaudhary Vikas Chaudhary added a comment -

            Reproduced on  6.6.0 build 7781

            Steps:

            1. create 2 node cluster 
            2. deploy 1 handler
            3. undeploy and delete it 
            4. rebalance in eventing node

            rebalance failed 

            Fix verified on 6.6.0 build 7839

            Show
            vikas.chaudhary Vikas Chaudhary added a comment - Reproduced on  6.6.0 build 7781 Steps: create 2 node cluster  deploy 1 handler undeploy and delete it  rebalance in eventing node rebalance failed  Fix verified on 6.6.0 build 7839

              People

              Assignee:
              ankit.prabhu Ankit Prabhu
              Reporter:
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty