Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40634

DEL_WITH_META fails: get_body_size: Failed to inflate data

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: couchbase-bucket
    • Labels:
    • Environment:

      Enterprise Edition 6.6.0 build 7883/6.6.0-7897

      Description

      – Create a replication between default bucket on source cluster to remove bucket on destination cluster. (Default bucket on source cluster has - compression - Active)
      – Following error is seen in babysitter.log and memcached.log. CPU on both the nodes is at 90%. Destination nodes have 2 nodes in cluster.

      Note - the same test on another cluster run smoothly without the error above.

      2020-07-26T20:55:59.064767-07:00 ERROR 319: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":402,"cas":0,"datatype":["Snappy","Xattr"],"extlen":24,"key":"<ud>9DC5E3F5-22_1536349</ud>","keylen":19,"magic":"ClientRequest","opaque":251658240,"opcode":"DEL_WITH_META","vbucket":179},"refcount":0}] - closing connection ([ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]): get_body_size: Failed to inflate data
      2020-07-26T20:55:59.065379-07:00 WARNING (remote) Invalid streaming connection
      

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          ritam.sharma Ritam Sharma created issue -
          ritam.sharma Ritam Sharma made changes -
          ritam.sharma Ritam Sharma made changes -
          Link to Log File, atop/blg, CBCollectInfo, Core dump XDCR Destination cluster -
          https://uploads.couchbase.com/MB-40634/collectinfo-2020-07-27T044525-ns_1%40172.23.105.111.zip
          https://uploads.couchbase.com/MB-40634/collectinfo-2020-07-27T044525-ns_1%40172.23.105.168.zip
          XDCR Destination cluster -
          https://uploads.couchbase.com/MB-40634/collectinfo-2020-07-27T044525-ns_1%40172.23.105.111.zip
          https://uploads.couchbase.com/MB-40634/collectinfo-2020-07-27T044525-ns_1%40172.23.105.168.zip

          XDCR Source Cluster:
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.137.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.155.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.157.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.5.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.61.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.70.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.87.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.104.88.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.106.100.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.106.188.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.108.103.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.120.245.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.96.148.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.96.251.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.96.252.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.96.253.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.96.95.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.97.119.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.97.121.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.97.122.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.97.239.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.97.242.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.98.135.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.99.20.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.99.21.zip
          https://uploads.couchbase.com/MB-40634-source/collectinfo-2020-07-27T044605-ns_1%40172.23.99.25.zip
          Hide
          trond Trond Norbye added a comment -

          From the log it looks like we're trying to inflate an item which claims to be compressed and contains xattrs, but we fail to inflate it and my guess is that the document isn't compressed with snappy.

          Could this be introduced with the changes we did for transactions?

          Show
          trond Trond Norbye added a comment - From the log it looks like we're trying to inflate an item which claims to be compressed and contains xattrs, but we fail to inflate it and my guess is that the document isn't compressed with snappy. Could this be introduced with the changes we did for transactions?
          Hide
          paolo.cocchi Paolo Cocchi added a comment -

          Trond Norbye Yes, related to the changes for CreateAsDeleted.
          The problem is that at EPE::deleteWithMeta (and before calling down to get_body_size()) we already inflate the payload but we miss to clear the snappy bit in the datatype.
          Daniel Owen Assigning to me for fixing.

          Show
          paolo.cocchi Paolo Cocchi added a comment - Trond Norbye Yes, related to the changes for CreateAsDeleted. The problem is that at EPE::deleteWithMeta (and before calling down to get_body_size()) we already inflate the payload but we miss to clear the snappy bit in the datatype. Daniel Owen Assigning to me for fixing.
          paolo.cocchi Paolo Cocchi made changes -
          Assignee Daniel Owen [ owend ] Paolo Cocchi [ paolo.cocchi ]
          ritam.sharma Ritam Sharma made changes -
          Description XDCR destination cluster memcached logs and babysitter.log

          -- Create a replication between default bucket on source cluster to remove bucket on destination cluster.
          -- Following error is seen in babysitter.log and memcached.log. CPU on both the nodes is at 90%. Destination nodes have 2 nodes in cluster.

          Note - the same test on another cluster run smoothly without the error above.

          {noformat}
          2020-07-26T20:55:59.064767-07:00 ERROR 319: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":402,"cas":0,"datatype":["Snappy","Xattr"],"extlen":24,"key":"<ud>9DC5E3F5-22_1536349</ud>","keylen":19,"magic":"ClientRequest","opaque":251658240,"opcode":"DEL_WITH_META","vbucket":179},"refcount":0}] - closing connection ([ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]): get_body_size: Failed to inflate data
          2020-07-26T20:55:59.065379-07:00 WARNING (remote) Invalid streaming connection
          {noformat}
          -- Create a replication between default bucket on source cluster to remove bucket on destination cluster. (Default bucket on source cluster has - compression - Active)
          -- Following error is seen in babysitter.log and memcached.log. CPU on both the nodes is at 90%. Destination nodes have 2 nodes in cluster.

          Note - the same test on another cluster run smoothly without the error above.

          {noformat}
          2020-07-26T20:55:59.064767-07:00 ERROR 319: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":402,"cas":0,"datatype":["Snappy","Xattr"],"extlen":24,"key":"<ud>9DC5E3F5-22_1536349</ud>","keylen":19,"magic":"ClientRequest","opaque":251658240,"opcode":"DEL_WITH_META","vbucket":179},"refcount":0}] - closing connection ([ 172.23.108.103:44786 - 172.23.105.111:11210 (<ud>Administrator</ud>) ]): get_body_size: Failed to inflate data
          2020-07-26T20:55:59.065379-07:00 WARNING (remote) Invalid streaming connection
          {noformat}
          paolo.cocchi Paolo Cocchi made changes -
          Triage Untriaged [ 10351 ] Triaged [ 10350 ]
          Hide
          ritam.sharma Ritam Sharma added a comment -

          it also blocks system testing to complete for cluster1, where the issue is seen for almost every run.

          Show
          ritam.sharma Ritam Sharma added a comment - it also blocks system testing to complete for cluster1, where the issue is seen for almost every run.
          drigby Dave Rigby made changes -
          Link This issue blocks MB-38724 [ MB-38724 ]
          drigby Dave Rigby made changes -
          Assignee Paolo Cocchi [ paolo.cocchi ] Ritam Sharma [ ritam.sharma ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.0-7898 contains kv_engine commit b08424f with commit message:
          MB-40634: Update datatype when decompressing the payload at DelWithMeta

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7898 contains kv_engine commit b08424f with commit message: MB-40634 : Update datatype when decompressing the payload at DelWithMeta
          drigby Dave Rigby made changes -
          Summary memcached log - exception occurred in runloop during packet execution DEL_WITH_META fails: get_body_size: Failed to inflate data
          drigby Dave Rigby made changes -
          Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
          Hide
          ritam.sharma Ritam Sharma added a comment -

          Test run started on centos longevity on 6.6.0-7898

          Show
          ritam.sharma Ritam Sharma added a comment - Test run started on centos longevity on 6.6.0-7898
          ritam.sharma Ritam Sharma made changes -
          VERIFICATION STEPS Validated with system test run for build 6.6.0-7898. Destination cluster does not have any error mentioned in the defect.
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-2807 contains kv_engine commit b08424f with commit message:
          MB-40634: Update datatype when decompressing the payload at DelWithMeta

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2807 contains kv_engine commit b08424f with commit message: MB-40634 : Update datatype when decompressing the payload at DelWithMeta

            People

            Assignee:
            ritam.sharma Ritam Sharma
            Reporter:
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty