Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40634

DEL_WITH_META fails: get_body_size: Failed to inflate data

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: couchbase-bucket
    • Labels:
    • Environment:

      Enterprise Edition 6.6.0 build 7883/6.6.0-7897

      Description

      – Create a replication between default bucket on source cluster to remove bucket on destination cluster. (Default bucket on source cluster has - compression - Active)
      – Following error is seen in babysitter.log and memcached.log. CPU on both the nodes is at 90%. Destination nodes have 2 nodes in cluster.

      Note - the same test on another cluster run smoothly without the error above.

      2020-07-26T20:55:59.064767-07:00 ERROR 319: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":402,"cas":0,"datatype":["Snappy","Xattr"],"extlen":24,"key":"<ud>9DC5E3F5-22_1536349</ud>","keylen":19,"magic":"ClientRequest","opaque":251658240,"opcode":"DEL_WITH_META","vbucket":179},"refcount":0}] - closing connection ([ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]): get_body_size: Failed to inflate data
      		 
      2020-07-26T20:55:59.065379-07:00 WARNING (remote) Invalid streaming connection

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          trond Trond Norbye added a comment -

          From the log it looks like we're trying to inflate an item which claims to be compressed and contains xattrs, but we fail to inflate it and my guess is that the document isn't compressed with snappy.

          Could this be introduced with the changes we did for transactions?

          Show
          trond Trond Norbye added a comment - From the log it looks like we're trying to inflate an item which claims to be compressed and contains xattrs, but we fail to inflate it and my guess is that the document isn't compressed with snappy. Could this be introduced with the changes we did for transactions?
          Hide
          Permalink
          paolo.cocchi Paolo Cocchi added a comment -

          Trond Norbye Yes, related to the changes for CreateAsDeleted.
          The problem is that at EPE::deleteWithMeta (and before calling down to get_body_size()) we already inflate the payload but we miss to clear the snappy bit in the datatype.
          Daniel Owen Assigning to me for fixing.

          Show
          paolo.cocchi Paolo Cocchi added a comment - Trond Norbye Yes, related to the changes for CreateAsDeleted. The problem is that at EPE::deleteWithMeta (and before calling down to get_body_size()) we already inflate the payload but we miss to clear the snappy bit in the datatype. Daniel Owen Assigning to me for fixing.
          Hide
          Permalink
          ritam.sharma Ritam Sharma added a comment -

          it also blocks system testing to complete for cluster1, where the issue is seen for almost every run.

          Show
          ritam.sharma Ritam Sharma added a comment - it also blocks system testing to complete for cluster1, where the issue is seen for almost every run.
          Hide
          Permalink
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.0-7898 contains kv_engine commit b08424f with commit message:
          MB-40634: Update datatype when decompressing the payload at DelWithMeta

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7898 contains kv_engine commit b08424f with commit message: MB-40634 : Update datatype when decompressing the payload at DelWithMeta
          Hide
          Permalink
          ritam.sharma Ritam Sharma added a comment -

          Test run started on centos longevity on 6.6.0-7898

          Show
          ritam.sharma Ritam Sharma added a comment - Test run started on centos longevity on 6.6.0-7898
          Hide
          Permalink
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-2807 contains kv_engine commit b08424f with commit message:
          MB-40634: Update datatype when decompressing the payload at DelWithMeta

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2807 contains kv_engine commit b08424f with commit message: MB-40634 : Update datatype when decompressing the payload at DelWithMeta

            People

            Assignee:
            ritam.sharma Ritam Sharma
            Reporter:
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty